Apparatuses and methods for navigation in and local segmentation extension of anatomical treelike structures

ABSTRACT

A local extension method for segmentation of anatomical treelike structures includes receiving an initial segmentation of 3D image data including an initial treelike structure. A target point in the 3D image data is defined, and a region of interest based on the target point is extracted to create a sub-image. Highly tubular voxels are detected in the sub-image, and a spillage-constrained region growing is performed using the highly tubular voxels as seed points. Connected components are extracted from the results of the region growing. The extracted components are pruned to discard components not likely to be connected to the initial treelike structure, keeping only candidate components likely to be a valid sub-tree of the initial treelike structure. The candidate components are connected to the initial treelike structure, thereby extending the initial segmentation in the region of interest.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application claiming priority to U.S.application Ser. No. 15/952,189, filed on Apr. 12, 2018 and issued asU.S. Pat. No. 10,643,333 on May 5, 2020; the entire content of which isincorporated herein by reference.

FIELD

The present disclosure relates generally to medical devices, andparticularly to apparatuses and methods associated with a range ofimage-guided medical procedures for detecting, sampling, staging, and/ortreating target tissues, including medical imaging and image processingmethods used therefore.

BACKGROUND

Image-guided surgery, also known as image-guided intervention, enhancesa physician's ability to locate instruments within anatomy during amedical procedure. Image-guided surgery can include 2-dimensional (2D),3-dimensional (3D), and 4-dimensional (4D) applications. The fourthdimension of image-guided surgery can include multiple parameters eitherindividually or together such as time, motion, electrical signals,pressure, airflow, blood flow, respiration, heartbeat, and other patientmeasured parameters.

Practitioners often use 3D medical imaging (e.g., CT, MRI) methods toassess a patient's anatomy and for use during treatment (e.g., duringimage-guided surgery). In order to access a target area (such as a node,nodule, tumor, or other target), the practitioner must navigate throughthe patient's anatomy to the target area. As disclosed and described inU.S. Ser. Nos. 13/817,730 and 15/290,822, the entireties of which areincorporated herein by reference, the medical images are segmented(e.g., to locate boundaries and objects within the images) to providenon-invasive information for use during a procedure, including but notlimited to navigating to the target area, determining the bestanatomical path to reach the target area, and localizing the medicalinstrument within the anatomy.

Although significant improvements have been made in these fields, a needremains for improved medical devices and procedures, including improvedsegmentation and image processing, for visualizing, accessing, locating,real-time confirming, sampling, and manipulating a target tissue orarea.

SUMMARY

One aspect of the disclosure is directed to a local extension method forsegmentation of anatomical treelike structures. The method includesreceiving an initial segmentation of 3D image data including an initialtreelike structure and defining a target point in the 3D image data. Aregion of interest is extracted based on the target point to create asub-image. Highly tubular voxels are detected in the sub-image, and aspillage-constrained region growing is performed using the highlytubular voxels as seed points. Connected components are extracted fromthe region growing step. The extracted components are pruned to discardcomponents not likely to be connected to the initial treelike structureand keep only candidate components likely to be a valid sub-tree of theinitial treelike structure. The candidate components are connected tothe initial treelike structure to extend the initial segmentation.

Another aspect of the disclosure is directed to a method includingreceiving 3D image data and segmenting an initial treelike structure inthe 3D image data to create an initial segmentation. A skeletonizationof the initial segmentation is created and voxels in the initialsegmentation are mask-labeled according to the skeletonization. A targetpoint in the initial segmentation is defined, and a region of interestis extracted based on the target point to create a sub-image. Thesub-image is smoothed, and the contrast in the sub-image is enhanced.Highly tubular voxels are detected in the sub-image, and aspillage-constrained region growing is performed using the highlytubular voxels as seed points. Connected components are extracted fromthe region growing results. A skeletonization of the extractedcomponents is created, and voxels in the extracted components aremask-labeled according to the skeletonization. The extracted componentsare pruned to discard components not likely to be connected to theinitial treelike structure and keep only candidate components likely tobe a valid sub-tree of the initial treelike structure. The candidatecomponents are connected to the initial treelike structure to extend theinitial segmentation.

Another aspect of the disclosure is directed to a local extension methodfor segmentation of anatomical treelike structures. The method includesreceiving an initial segmentation of 3D image data including an initialtreelike structure and defining a target point in the 3D image data. Aregion of interest is extracted based on the target point to create asub-image. Highly tubular voxels are detected in the sub-image, and aspillage-constrained region growing is performed using the highlytubular voxels as seed points. Connected components are extracted fromthe region growing step. The extracted components are pruned to discardcomponents not likely to be connected to the initial treelike structureand keep only candidate components likely to be a valid sub-tree of theinitial treelike structure. The candidate components are connected tothe initial treelike structure to extend the initial segmentation.

Another aspect of the disclosure is directed to a method includingreceiving 3D image data and segmenting an initial treelike structure inthe 3D image data to create an initial segmentation. A skeletonizationof the initial segmentation is created and voxels in the initialsegmentation are mask-labeled according to the skeletonization. A targetpoint in the initial segmentation is defined, and a region of interestis extracted based on the target point to create a sub-image. Thesub-image is smoothed, and the contrast in the sub-image is enhanced.Highly tubular voxels are detected in the sub-image, and aspillage-constrained region growing is performed using the highlytubular voxels as seed points. Connected components are extracted fromthe region growing results. A skeletonization of the extractedcomponents is created, and voxels in the extracted components aremask-labeled according to the skeletonization. The extracted componentsare pruned to discard components not likely to be connected to theinitial treelike structure and keep only candidate components likely tobe a valid sub-tree of the initial treelike structure. The candidatecomponents are connected to the initial treelike structure to extend theinitial segmentation.

Yet another aspect of the disclosure is directed to a local extensionmethod for segmentation of anatomical treelike structures. The methodincludes receiving an initial segmentation of image data including aninitial treelike structure and extracting a region of interest from theinitial segmentation to create a sub-image. Highly tubular voxels aredetected in the sub-image, and a spillage-constrained region growing isperformed using the highly tubular voxels as seed points. Connectedcomponents are extracted from the results of the region growing step.The extracted components are pruned to discard components not likely tobe connected to the initial treelike structure and keep only candidatecomponents likely to be a valid sub-tree of the initial treelikestructure. The candidate components are connected to the initialtreelike structure to extend the initial segmentation.

Still another aspect of the disclosure is directed to a method ofextending a segmentation of an image using information from a navigationsystem. The method includes tracking, with the navigation system, atleast one of a traveled path and a position of an imaging devicerelative to an initial segmentation of 3D image data including aninitial treelike structure. 2D image data comprising at least one 2Dimage is captured with the imaging device. A point from the 2D imagedata corresponding to a potential airway structure is obtained by thenavigation system. The initial segmentation of 3D image data is extendedby the navigation system using the point obtained from the 2D imagedata.

Another aspect of the disclosure is directed to a method of extending asegmentation of an image using information from a navigation system. Themethod includes tracking, with the navigation system, at least one of atraveled path, a position, and a trajectory of a navigated instrumentrelative to an initial segmentation of 3D image data including aninitial treelike structure. The initial segmentation of the 3D imagedata is extended using the tracked traveled path, position, ortrajectory of the navigated instrument. Extending the initialsegmentation includes extracting a region of interest from the 3D imagedata to create a sub-image. Connected components based onspillage-constrained region growing using seed points from the sub-imageare extracted. The extracted components are pruned to discard componentsnot likely to be connected to the initial treelike structure and keeponly candidate components likely to be a valid sub-tree of the initialtreelike structure. The candidate components are connected to theinitial treelike structure to extend the segmentation.

Another aspect of the disclosure is directed to a method of extending asegmentation of an image using navigated image data from a navigationsystem. The method includes tracking, with the navigation system, atleast one of a traveled path and a position of an imaging devicerelative to an initial segmentation of 3D image data including aninitial treelike structure. Navigated image data comprising image dataincluding at least one 2D or 3D image is captured with the imagingdevice. A point from the navigated image data corresponding to apotential airway structure is obtained by the navigation system. Theinitial segmentation of 3D image data is extended by the navigationsystem using the point obtained from the navigated image data.

Another aspect of the disclosure is directed to a method of extending asegmentation of an image using navigation data from a navigation system.The method includes tracking, with the navigation system, at least oneof a traveled path, a position, and a trajectory of a navigatedinstrument relative to an initial segmentation of 3D image dataincluding an initial treelike structure to create navigation data. Theinitial segmentation of the 3D image data is extended using thenavigation data. Extending the initial segmentation includes extractinga region of interest from the 3D image data to create a sub-image andextracting connected components based on spillage-constrained regiongrowing using seed points from the sub-image. The extracted componentsare pruned to discard components not likely to be connected to theinitial treelike structure and keep only candidate components likely tobe a valid sub-tree of the initial treelike structure. The candidatecomponents are connected to the initial treelike structure to extend theinitial segmentation.

Other objects and features will be in part apparent and in part pointedout hereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a left perspective view of a patient tracking device on apatient according to an embodiment of the invention;

FIG. 2 is a schematic illustration of an image analysis system accordingto an embodiment of the invention;

FIG. 3 is a schematic illustration of a navigation system according toan embodiment of the invention;

FIG. 4 is a graphical representation illustrating the function of thepatient tracking device according to an embodiment of the invention;

FIG. 5A is an illustration of a patient being imaged using an imagingdevice according to an embodiment of the invention;

FIG. 5B is an illustration of a patient being imaged using an imagingdevice according to an embodiment of the invention;

FIG. 5C is a schematic illustration of an image dataset according to anembodiment of the invention;

FIG. 6A is a schematic illustration of an inspiration 3D airway modelaccording to an embodiment of the invention;

FIG. 6B is a schematic illustration of an expiration 3D airway modelaccording to an embodiment of the invention;

FIG. 6C is a schematic illustration of a hybrid “Inspiration-Expiration”3D airway model according to an embodiment of the invention;

FIG. 7 is a front perspective view of a hybrid “Inspiration-Expiration”3D airway model according to an embodiment of the invention;

FIG. 8 is a schematic illustrating vector distances of the patienttracking device according to an embodiment of the invention;

FIG. 9A is a schematic illustrating vector distances from a localizationelement on the patient tracking device according to an embodiment of theinvention;

FIG. 9B is a schematic illustrating vector distances from an imagedataset according to an embodiment of the invention;

FIG. 10 is a flowchart illustrating a method according to an embodimentof the invention;

FIG. 11 is a left side view of a steerable catheter according to anembodiment of the invention;

FIG. 11A is a left partial section view of a steerable catheteraccording to an embodiment of the invention;

FIG. 12A is a left partial cut away view of a steerable catheteraccording to an embodiment of the invention;

FIG. 12B is a left partial cut away view of a steerable catheteraccording to an embodiment of the invention;

FIG. 13 illustrates a population of images which may be displayed on adisplay of a navigation system according to an embodiment of theinvention;

FIG. 14 is a flowchart illustrating a method of local extension for asegmentation according to an embodiment;

FIG. 15 is a flowchart illustrating a method of defining a region ofinterest according to an embodiment;

FIG. 16 is a flowchart illustrating a method of pruning extractedcomponents according to an embodiment;

FIG. 17 is a flowchart illustrating a method of connecting components toan initial treelike structure according to an embodiment;

FIG. 18 is an image for navigating to a target, illustrating an initialpath to a target based on an initial segmentation;

FIG. 19 is an image for navigating to a target, illustrating a revisedpath to a target based on an extended segmentation; and

FIG. 20 is a flowchart illustrating a method of using 2D image data toextend a segmentation according to an embodiment.

Like reference numerals indicate corresponding parts throughout theseveral views of the drawings.

DETAILED DESCRIPTION

The accompanying Figures and this description depict and describeembodiments of a navigation system (and related methods and devices) inaccordance with the present invention, and features and componentsthereof. It should also be noted that any references herein to front andback, right and left, top and bottom and upper and lower are intendedfor convenience of description, not to limit the present invention orits components to any one positional or spatial orientation.

Those of skill in the art will appreciate that in the detaileddescription below, certain well known components and assembly techniqueshave been omitted so that the present methods, apparatuses, and systemsare not obscured in unnecessary detail.

Embodiments of the present disclosure are directed to extending branchesof segmented anatomical treelike structures in 3D image data. As usedherein, the term 3D image data refers to any type of 3-dimensionalimaging modalities, including but not limited to computed tomography,fused computed tomography/positron emission tomography, magneticresonance imaging, positron emission tomography, single photon emissioncomputed tomography, 3D ultrasound, 3D fluoroscopy, or any other 3Dimaging modality.

With larger volumes of patients expected to obtain lung cancerscreening, obtaining definitive diagnoses may avoid numerous unneededlung resections as about only 4% of patients from lung cancer screeningare typically found to have a malignancy. However, peripheral targettissues (e.g., nodule, lesion, lymph node, tumor, etc.) that are smallerthan 2 cm in size still present a difficult problem to solve. Typicalbronchoscopes that are designed mainly for central airway inspectionwill be limited to the extent they can travel due to their largediameters before becoming wedged in the airway of the patient. Thus, toaffect the 5 and 10 year survival rate of patient's that have targettissues which may be less than 2 cm in size, the apparatuses and methodsas described herein allow for enhanced target tissue analysis forstaging, intercepting target tissues in the periphery of the lungs thatmay not be accessible via airways, obtaining larger and higher qualitytissue samples for testing, and provide a streamlined patient flow.Accordingly, the apparatuses and methods described herein enable aphysician or other healthcare professional to initially determine thelocation of a target tissue and to confirm the location of the targettissue. In one embodiment, a hybrid “Inspiration-Expiration” 3D modelmay be used to provide patient specific 4D respiratory models whichaddress peripheral respiratory motion. In certain patients, portions ofthe lungs including the upper lobes may move, on average, 15 mm betweeninspiration and expiration. Using a steerable catheter with an imagingdevice, such as a radial endobronchial ultrasound (EBUS) device insertedtherein, a physician or other healthcare professional can determine aconfirmed location of the target tissue. Additionally, apparatuses andmethods described herein enable a physician or other healthcareprofessional to transition to a percutaneous approach to the targettissue, if needed, or to access the target tissue endobronchially ifpossible. If the physician or other healthcare professional is unable toreach the target tissue for any reason, including but not limited to,the target tissue being below the surface of the airway (i.e.,sub-surface target tissue), no airway proximate the target tissue, thepathway to the target tissue is very tortuous, or larger or additionaltissue sample from a core biopsy is desired, the physician or otherhealthcare professional may insert navigated percutaneous needles to theconfirmed location of the target tissue. Thus, it will be understoodthat the apparatuses and methods described herein may be used tointercept target tissue(s) in the airway, on the wall of the airway, inthe wall of the airway, and/or beyond the wall of the airway. That is,the apparatuses and methods described herein may be used to intercepttarget tissue(s) not only inside the airway, but may intercept targettissue(s) and other anatomical structures inside and/or beyond the wallof the airway. Thus, in certain embodiments, sub-surface targettissue(s) may be intercepted.

As shown in FIG. 1 , an apparatus according to an embodiment of theinvention includes patient tracking device (PTD) 20 comprising two ormore markers 22 and two or more localization elements 24 proximatemarkers 22. Markers 22 are visible in images captured by an imagingdevice and the position and orientation (POSE) of localization elements24 may be tracked by a localization device in an image analysis systemand/or a navigation system. PTD 20 comprises a population of separatepads 26, 28, 30, each of which may include one or more markers 22 andlocalization elements 24 proximate markers 22. First and second pads 26,28 may each include one marker 22 and one localization element 24. Thirdpad 30 may include four markers 22 and four localization elements 24located proximate the periphery of third pad 30. Additionally, wires 32,34, 36 are used to connect localization elements 24 in each of first,second, and third pads 26, 28, 30 to image analysis system 50 (see FIG.2 ) and/or navigation system 70 (see FIG. 3 ). In alternativeembodiments, localization elements 24 may be wirelessly connected tonavigation system 70. FIG. 1 illustrates PTD 20 having six markers 22and six localization elements 24, but any number of two or more markers22 and localization elements 24 can be used. Patient tracking device(PTD) 20 can be coupled to a dynamic body such as, for example, aselected dynamic portion of the anatomy of a patient 10.

Markers 22 are constructed of a material that can be viewed on an image,such as, for example, X-ray images or CT images. In certain embodiments,markers 22 can be, for example, radiopaque such that they are visiblevia fluoroscopic imaging. In other embodiments, for example, markers 22may be echogenic such that they are visible via ultrasonic imaging. Inyet other embodiments, markers 22 may be both radiopaque and echogenic.In certain embodiments, for example, localization elements 24 comprisesix (6) degree of freedom (6DOF) electromagnetic coil sensors. In otherembodiments, localization elements 24 comprise five (5) degree offreedom (5DOF) electromagnetic coil sensors. In other embodiments,localization elements 24 comprise other localization devices such asradiopaque markers that are visible via fluoroscopic imaging andechogenic patterns that are visible via ultrasonic imaging. In yet otherembodiments, localization elements 24 can be, for example, infraredlight emitting diodes, and/or optical passive reflective markers.Localization elements 24 can also be, or be integrated with, one or morefiber optic localization (FDL) devices.

While PTD 20 is shown comprising a population of separate padscontaining markers 22 and localization elements 24, in certainembodiments, PTD 20 may comprise one pad containing markers 22 andlocalization elements 24. In another embodiment, for example, PTD 20 mayinclude markers 22 but not localization elements 24. In anotherembodiment, for example, PTD 20 may include localization elements 24 butnot markers 22. In various embodiments, markers 22 and localizationelements 24 can be the same device. In certain embodiments, for example,localization elements 24 may function or serve as markers 22. PTD 20 canbe a variety of different shapes and sizes. For example, in oneembodiment PTD 20 is substantially planar, such as in the form of a padthat can be disposed at a variety of locations on a patient's 10 body.PTD 20 can be coupled to patient 10 with adhesive, straps, hook andpile, snaps, or any other suitable coupling method. In anotherembodiment the PTD can be a catheter type device with a pigtail oranchoring mechanism that allows it to be attached to an internal organor along a vessel.

As described more fully elsewhere herein, an image analysis system isconfigured to receive image data associated with the dynamic bodygenerated during a pre-surgical or pre-procedural first time interval.The image data can include an indication of a position of each ofmarkers 22 for multiple instants in time during the first time interval.Then a navigation system can also receive position data associated withlocalization elements 24 during a second time interval in which asurgical procedure or other medical procedure is being performed. Thenavigation system can use the position data received from localizationelements 24 to determine a distance between the localization elements 24for a given instant in time during the second time interval. Thenavigation system can also use the image data to determine the distancebetween markers 22 for a given instant in time during the first timeinterval. The navigation system can then find a match between an imagewhere the distance between markers 22 at a given instant in time duringthe first time interval is the same or substantially the same as thedistance between localization elements 24 associated with those markers22 at a given instant in time during the medical procedure, or secondtime interval. Additionally, the navigation system can determine asequence of motion of the markers and match this sequence of motion tothe recorded motion of the markers over the complete procedure orsignificant period of time. Distance alone between the markers may notbe sufficient to match the patient space to image space in manyinstances, the system may also determine the direction the markers aremoving and the range and speed of this motion to find the appropriatesequence of motion for a complex signal or sequence of motion by thepatient.

A physician or other healthcare professional can use the images selectedby the navigation system during a medical procedure performed during thesecond time interval. For example, when a medical procedure is performedon a targeted anatomy of a patient, such as a heart or lung, thephysician may not be able to utilize an imaging device during themedical procedure to guide him to the targeted area within the patient.Accordingly, PTD 20 can be positioned or coupled to the patientproximate the targeted anatomy prior to the medical procedure, andpre-procedural images can be taken of the targeted area during a firsttime interval. Markers 22 of PTD 20 can be viewed with the image data,which can include an indication of the position of markers 22 during agiven path of motion of the targeted anatomy (e.g., the heart) duringthe first time interval. Such motion can be due, for example, toinspiration (i.e., inhaling) and expiration (i.e., exhaling) of thepatient, or due to the heart beating. During a medical procedure,performed during a second time interval, such as a procedure on a heartor lung, the navigation system receives data from localization elements24 associated with a position of localization elements 24 at a giveninstant in time during the medical procedure (or second time interval).The distance between selected pairs of markers 22 can be determined fromthe image data and the distance, range, acceleration, and speed betweencorresponding selected pairs of localization elements 24 can bedetermined based on the position and orientation (POSE) data for giveninstants in time. Accordingly, the range of motion and speed of markers22 can be calculated.

Because localization elements 24 are proximate the location of markers22, the distance between a selected pair of localization elements 24 canbe used to determine an intra-procedural distance between the pair ofcorresponding markers 22. An image from the pre-procedural image datataken during the first time interval can then be selected where thedistance between the pair of selected markers 22 in that imagecorresponds with or closely approximates the same distance determinedusing localization elements 24 at a given instant in time during thesecond time interval. This process can be done continuously during themedical procedure, producing simulated real-time, intra-proceduralimages illustrating the orientation and shape of the targeted anatomy asa catheter, sheath, needle, forceps, guidewire, fiducial deliverydevices, therapy device, or similar medical device(s) is/are navigatedto the targeted anatomy. Thus, during the medical procedure, thephysician can view selected image(s) of the targeted anatomy thatcorrespond to and simulate real-time movement of the anatomy. Inaddition, during a medical procedure being performed during the secondtime interval, such as navigating a catheter or other medical device orcomponent thereof to a targeted anatomy, the location(s) of alocalization element (e.g., an electromagnetic coil sensor) coupled tothe catheter during the second time interval can be superimposed on animage of a catheter. The superimposed image(s) of the catheter can thenbe superimposed on the selected image(s) from the first time interval,providing simulated real-time images of the catheter location relativeto the targeted anatomy. This process and other related methods aredescribed in U.S. Pat. No. 7,398,116, entitled Methods, Apparatuses, andSystems Useful in Conducting Image Guided Interventions, filed Aug. 26,2003, which is hereby incorporated by reference.

Referring now to FIGS. 2 and 3 , two systems which may be used duringimage guided surgery are described in detail. The first systemillustrated in FIG. 2 , is image analysis system 50. Image analysissystem 50 is used during generation of a population of images of patient10 during a first time interval, prior to a medical procedure beingperformed on patient 10. The second system, illustrated in FIG. 3 , isnavigation system 70. Navigation system 70 is used during a medicalprocedure performed on patient 10 during a second time interval. As willbe described, imaging system 50 and navigation system 70 may include, invarious embodiments, substantially similar or identical components.Accordingly, image analysis system 50 and navigation system 70 may beable to carry out substantially similar or identical functions. Incertain embodiments, image analysis system 50 and navigation system 70and may comprise a single system. In certain embodiments, for example,image analysis system 50 may also function or serve as a navigationsystem. In certain embodiments, for example, navigation system 70 mayalso function or serve as an image analysis system.

As shown in FIG. 2 , image analysis system 50 comprises a processor 52having memory component 54, input/output (I/O) component 58, andoptional localization device 56. Image analysis system 50 may alsooptionally include display 60, electromagnetic field generator 62,and/or user interface device(s) 64 (e.g., keyboard, mouse).

Image analysis system 50 further includes and/or is in datacommunication with imaging device 40. Imaging device 40 can be, forexample, a computed tomography (CT) device (e.g., respiratory-gated CTdevice, ECG-gated CT device), a magnetic resonance imaging (MRI) device(e.g., respiratory-gated MRI device, ECG-gated MRI device), an X-raydevice, a 2D or 3D fluoroscopic imaging device, and 2D, 3D or 4Dultrasound imaging devices, or any other suitable medical imagingdevice. In one embodiment, for example, imaging device 40 is a computedtomography-positron emission tomography (CT/PET) device that produces afused computed tomography-positron emission tomography (CT/PET) imagedataset. In the case of a two-dimensional imaging device, a populationof two-dimensional images may be acquired and then assembled intovolumetric data (e.g., three-dimensional (3D) image dataset) as is knownin the art using a two-dimensional to three-dimensional conversion.Pre-procedurally during a first time interval, imaging device 40 can beused to generate a population of images of patient 10 while PTD 20 iscoupled to patient 10; wherein the population of images depict theanatomy of patient 10. The anatomy, may include, but is not limited to,the lungs, heart, liver, kidneys, and/or other organs of patient 10. Thepopulation of images can be compiled into an image dataset. As statedabove, some or all markers 22 of PTD 20 are visible on the population ofimages and provide an indication of a position of some or all of markers22 during the first time interval. The position of markers 22 at giveninstants in time through a path of motion of patient 10 can beillustrated with the images.

Processor 52 of image analysis system 50 includes a processor-readablemedium storing code representing instructions to cause the processor 52to perform a process. Processor 52 can be, for example, a commerciallyavailable personal computer, or a less complex computing or processingdevice that is dedicated to performing one or more specific tasks. Forexample, processor 52 can be a terminal dedicated to providing aninteractive graphical user interface (GUI) on optional display 60.Processor 52, according to one or more embodiments of the invention, canbe a commercially available microprocessor. Alternatively, processor 52can be an application-specific integrated circuit (ASIC) or acombination of ASICs, which are designed to achieve one or more specificfunctions, or enable one or more specific devices or applications. Inyet another embodiment, processor 52 can be an analog or digitalcircuit, or a combination of multiple circuits.

Additionally, processor 52 can include memory component 54. Memorycomponent 54 can include one or more types of memory. For example,memory component 54 can include a read only memory (ROM) component and arandom access memory (RAM) component. Memory component 54 can alsoinclude other types of memory that are suitable for storing data in aform retrievable by processor 52. For example, electronicallyprogrammable read only memory (EPROM), erasable electronicallyprogrammable read only memory (EEPROM), flash memory, as well as othersuitable forms of memory can be included within the memory component.Processor 52 can also include a variety of other components, such as forexample, coprocessors, graphic processors, etc., depending upon thedesired functionality of the code.

Processor 52 can store data in memory component 54 or retrieve datapreviously stored in memory component 54. The components of processor 52can communicate with devices external to processor 52 by way ofinput/output (I/O) component 58. According to one or more embodiments ofthe invention, I/O component 58 includes a variety of suitablecommunication interfaces. For example, I/O component 58 can include, forexample, wired connections, such as standard serial ports, parallelports, universal serial bus (USB) ports, S-video ports, local areanetwork (LAN) ports, small computer system interface (SCSI) ports, andso forth. Additionally, I/O component 58 can include, for example,wireless connections, such as infrared ports, optical ports, Bluetooth®wireless ports, wireless LAN ports, or the like. Embodiments of imageanalysis system 50 which include display 60, electromagnetic fieldgenerator 62, and/or user interface device(s) 64, such componentscommunicate with processor 52 via I/O component 58.

Processor 52 can be connected to a network, which may be any form ofinterconnecting network including an intranet, such as a local or widearea network, or an extranet, such as the World Wide Web or theInternet. The network can be physically implemented on a wireless orwired network, on leased or dedicated lines, including a virtual privatenetwork (VPN).

As stated above, processor 52 receives the population of images fromimaging device 40. Processor 52 identifies the position of selectedmarkers 22 within the image data or voxel space using varioussegmentation techniques, such as Hounsfield unit thresholding,convolution, connected component, or other combinatory image processingand segmentation techniques. Segmentation methods for identifyinganatomical structure and for use during navigation are described indetail herein below. Processor 52 determines a distance and directionbetween the position of any two markers 22 during multiple instants intime during the first time interval, and stores the image data, as wellas the position and distance data, within memory component 54. Multipleimages can be produced providing a visual image at multiple instants intime through the path of motion of the dynamic body.

As stated above, processor 52 can optionally include a receiving deviceor localization device 56 for tracking the location of localizationelements 24 of PTD 20, as described more fully elsewhere herein. Bytracking localization elements 24 associated with PTD 20 when thepopulation of images are generated by imaging device 40, the populationof images may be gated. That is, image analysis system 50 determines therespiratory phase at which the population of images were generated, andthis information may be stored in an image dataset and/or in anotherdata store in memory component 54.

In general, image analysis system 50 may comprise any tracking systemtypically employed in image guided surgery, including but not limitedto, an electromagnetic tracking system. An example of a suitableelectromagnetic tracking subsystem is the AURORA electromagnetictracking system, commercially available from Northern Digital Inc.(Waterloo, Ontario Canada). In one embodiment, image analysis system 50may include an electromagnetic tracking system, typically comprising anelectromagnetic (EM) field generator 62 that emits a series ofelectromagnetic fields designed to engulf patient 10, and localizationelements 24 coupled to PTD 20. In certain embodiments, for example,localization elements 24 are electromagnetic coils that receive aninduced voltage from electromagnetic (EM) field generator 62, whereinthe induced voltage is monitored and translated by localization device56 into a coordinate position of localization elements 24. In certainembodiments, localization elements 24 are electrically coupled totwisted pair conductors to provide electromagnetic shielding of theconductors. This shielding prevents voltage induction along theconductors when exposed to the magnetic flux produced by theelectromagnetic field generator.

Accordingly, localization device 56 can be, for example, an analog todigital converter that measures voltages induced onto localizationelements 24 in the field generated by EM field generator 62; creates adigital voltage reading; and maps that voltage reading to a metricpositional measurement based on a characterized volume of voltages tomillimeters from electromagnetic field generator 62. Position dataassociated with localization elements 24 can be transmitted or sent tolocalization device 56 continuously during imaging of patient 10 duringthe first time interval. Thus, the position of localization elements 24can be captured at given instants in time during the first timeinterval. Because localization elements 24 are proximate markers 22,localization device 56 uses the position data of localization elements24 to deduce coordinates or positions associated with markers 22 duringthe first time interval. The distance, range, acceleration, and speedbetween one or more selected pairs of localization elements 24 (andcorresponding markers 22) is then determined and various algorithms areused to analyze and compare the distance between selected elements 24 atgiven instants in time, to the distances between and orientation amongcorresponding markers 22 observed in a population of pre-proceduralimages.

As shown in FIG. 3 , navigation system 70 comprises a processor 72having memory component 74, input/output (I/O) component 78, andlocalization device 76. Navigation system 70 also includes display 80,electromagnetic field generator 82, and/or user interface device(s) 84(e.g., keyboard, mouse). In certain embodiments, navigation system 50further includes and/or is in data communication with imaging device 40(see FIG. 2 ).

Processor 72 of navigation system 70 includes a processor-readablemedium storing code representing instructions to cause the processor 72to perform a process. Processor 72 can be, for example, a commerciallyavailable personal computer, or a less complex computing or processingdevice that is dedicated to performing one or more specific tasks. Forexample, processor 72 can be a terminal dedicated to providing aninteractive graphical user interface (GUI) on optional display 80.Processor 72, according to one or more embodiments of the invention, canbe a commercially available microprocessor. Alternatively, processor 72can be an application-specific integrated circuit (ASIC) or acombination of ASICs, which are designed to achieve one or more specificfunctions, or enable one or more specific devices or applications. Inyet another embodiment, processor 72 can be an analog or digitalcircuit, or a combination of multiple circuits.

Additionally, processor 72 can include memory component 74. Memorycomponent 74 can include one or more types of memory. For example,memory component 74 can include a read only memory (ROM) component and arandom access memory (RAM) component. Memory component 74 can alsoinclude other types of memory that are suitable for storing data in aform retrievable by processor 72. For example, electronicallyprogrammable read only memory (EPROM), erasable electronicallyprogrammable read only memory (EEPROM), flash memory, as well as othersuitable forms of memory can be included within the memory component.Processor 72 can also include a variety of other components, such as forexample, coprocessors, graphic processors, etc., depending upon thedesired functionality of the code.

Processor 72 can store data in memory component 74 or retrieve datapreviously stored in memory component 74. The components of processor 72can communicate with devices external to processor 72 by way ofinput/output (I/O) component 78. According to one or more embodiments ofthe invention, I/O component 78 includes a variety of suitablecommunication interfaces. For example, I/O component 78 can include, forexample, wired connections, such as standard serial ports, parallelports, universal serial bus (USB) ports, S-video ports, local areanetwork (LAN) ports, small computer system interface (SCSI) ports, andso forth. Additionally, I/O component 78 can include, for example,wireless connections, such as infrared ports, optical ports, Bluetooth®wireless ports, wireless LAN ports, or the like. Additionally, display80, electromagnetic field generator 82, and/or user interface device(s)84, communicate with processor 72 via I/O component 78.

Processor 72 can be connected to a network, which may be any form ofinterconnecting network including an intranet, such as a local or widearea network, or an extranet, such as the World Wide Web or theInternet. The network can be physically implemented on a wireless orwired network, on leased or dedicated lines, including a virtual privatenetwork (VPN).

In general, navigation system 70 may comprise any tracking systemtypically employed in image guided surgery, including but not limitedto, an electromagnetic tracking system. An example of a suitableelectromagnetic tracking subsystem is the AURORA electromagnetictracking system, commercially available from Northern Digital Inc.(Waterloo, Ontario Canada). In one embodiment, navigation system 70 mayinclude an electromagnetic tracking system, typically comprising anelectromagnetic (EM) field generator 82 that emits a series ofelectromagnetic fields designed to engulf patient 10, and localizationelements 24 coupled to PTD 20. In certain embodiments, for example,localization elements 24 are electromagnetic coils that receive aninduced voltage from electromagnetic (EM) field generator 82, whereinthe induced voltage is monitored and translated by localization device76 into a coordinate position of localization elements 24. In certainembodiments, localization elements 24 are electrically coupled totwisted pair conductors to provide electromagnetic shielding of theconductors. This shielding prevents voltage induction along theconductors when exposed to the magnetic flux produced by theelectromagnetic field generator.

Accordingly, localization device 76 may be, for example, an analog todigital converter that measures voltages induced onto localizationelements 24 in the field generated by EM field generator 82; creates adigital voltage reading; and maps that voltage reading to a metricpositional measurement based on a characterized volume of voltages tomillimeters from electromagnetic field generator 82. Position dataassociated with localization elements 24 may be transmitted or sent tolocalization device 76 continuously during the medical procedureperformed during the second time interval. Thus, the position oflocalization elements 24 may be captured at given instants in timeduring the second time interval. Because localization elements 24 areproximate markers 22, localization device 76 uses the position data oflocalization elements 24 to deduce coordinates or positions associatedwith markers 22 during the second time interval. The distance, range,acceleration, and speed between one or more selected pairs oflocalization elements 24 (and corresponding markers 22) is thendetermined and various algorithms are used to analyze and compare thedistance between selected elements 24 at given instants in time, to thedistances between and orientation among corresponding markers 22observed in a population of pre-procedural images.

Because localization elements 24 of PTD 20 may be tracked continuouslyduring the first and/or second time intervals, a sequence of motion ofPTD 20 that represents the motion of an organ of patient 10 or thepatient's 10 respiratory cycle may be collected. As patient 10 inhalesand exhales, the individual localization elements 24 of PTD 20 will moverelative to one another. That is, as patient 10 inhales, the distancebetween some or all of localization elements 24 of PTD 20 may increase.Conversely, as patient 10 exhales, the distance between some or all oflocalization elements 24 of PTD 20 may decrease. The sequence of motionof localization elements 24 is tracked by image analysis system 50and/or navigation system 70 and image analysis system 50 and/ornavigation system 70 derives a respiratory signal based on the positionsof localization elements 24 during the respiratory cycle of patient 10.The sequence of motion may then be analyzed to find unique similarpoints within the image dataset and images within the image dataset maybe grouped.

According to one particular embodiment, the respiratory signal derivedfrom PTD 20 is used to gate the localization information of a medicaldevice in the airway of patient 10. In other embodiments, therespiratory signal derived from PTD 20 is used during the first timeinterval to gate the population of images generated by imaging device40. Using PTD 20 to derive a respiratory signal may assist indetermining multiple airway models, for example, by performing a bestfit of the real-time patient airway model to the image dataset to derivethe optimal registration and gated period in the patient's respiratorycycle. Additionally or alternatively, the respiratory signal may bederived from devices other than PTD 20 that are known in the art formeasuring the respiratory cycle of a patient. In certain embodiments,for example, a device that records the resistance between two locationson the patient may be used to measure the respiratory cycle. Forexample, such a device is similar to a variable potentiometer in thatthe resistance of the patient changes between two fixed points as thepatient inhales and exhales. Thus, the resistance may be measured tocreate a respiratory signal. In other embodiments, a spirometer may beused to measure the respiratory cycle. In yet other embodiments, acardiac signal may be used to gate the localization information of amedical device in the airway of patient 10. It will be understood thatany type of device for generating a cardiac signal may be used,including, but not limited to an ECG device, PTD 20, etc.

FIG. 4 is a schematic illustration indicating how markers 22 of PTD 20can move and change orientation and shape during movement of patient 10.The graph is one example of how the lung volume can change duringinhalation (inspiration) and exhalation (expiration) of patient 10. Thecorresponding changes in shape and orientation of PTD 20 duringinhalation and exhalation are also illustrated. The six markers 22 shownin FIG. 1 are schematically represented and labeled a, b, c, d, e, andf. As described above, a population of images of PTD 20 may be takenduring a first time interval. The population of images include anindication of relative position of one or more markers 22; that is, oneor more markers 22 are visible in the images, and the position of eachmarker 22 is then observed over a period of time. A distance between anytwo markers 22 may then be determined for any given instant of timeduring the first time interval. For example, a distance X betweenmarkers a and b is illustrated, and a distance Y between markers b and fis illustrated. These distances may be determined for any given instantin time during the first time interval from an associated image thatillustrates the position and orientation of markers 22. As illustrated,during expiration of patient 10 at times indicated as A and C, thedistance X is smaller than during inspiration of patient 10, at the timeindicated as B. Likewise, the distance Y is greater during inspirationthan during expiration. The distance between any pair of markers 22 maybe determined and used in the processes described herein. Thus, theabove embodiments are merely examples of possible pair selections. Forexample, a distance between a position of marker e and a position ofmarker b may be determined. In addition, multiple pairs or only one pairmay be selected for a given procedure.

FIGS. 5A and 5B illustrate the generation of a population of imagesduring a first time interval using imaging device 40, PTD 20, andoptionally electromagnetic field generator 62 of image analysis system50. In FIG. 5A, patient 10 inhales and patient 10 is scanned usingimaging device 40 which generates a population of images 402 of theanatomy of patient 10 and markers 22 at inspiration. As shown, patient10 may place their arms above their head as they inhale, and this may beconsidered a total lung capacity (TLC) scan. In FIG. 5B, patient 10exhales and patient 10 is scanned using imaging device 40 whichgenerates a population of images 404 of the anatomy of patient 10 andmarkers 22 at expiration. As shown, patient 10 may place their armsbelow their head, and this may be considered a functional residualcapacity (FRC) scan. The Functional Residual Capacity is the lung volumeat the end of a normal expiration, when the muscles of respiration arecompletely relaxed. At FRC (and typically at FRC only), the tendency ofthe lungs to collapse is exactly balanced by the tendency of the chestwall to expand. In various embodiments, the population of images 402,404 may be two-dimensional (2D) images. In other embodiments, forexample, the population of images 402, 404 may be three-dimensional (3D)images. Additionally, the population of images 402, 404 may berespiratory gated by tracking the location of localization elements 24of PTD 20 by image analysis system 50 and/or navigation system 70 usingEM field generator 62, 82 during image generation. In other embodiments,for example, the population of images 402, 404 may be gated using anytype of device known for generating a physiological signal for gating.

In various embodiments, for example, instead of patient 10 holding aninspiration or expiration state, a cine loop of images may be generatedin conjunction with the patient's respiratory cycle information from PTD20. Thus, the cine loop comprises a population of images generated frominspiration to expiration where the population of images are gated tothe respiratory cycle of patient 10 using PTD 20. This can serve tolimit registration point selection, in order to be consistent with thepatient's respiratory cycle that a 3D dataset such as CT, MR, or PET hasacquired. This technique advantageously maximizes registration accuracy,a major flaw in conventional systems in the prior art.

As described above, imaging device 40 is in data communication withimage analysis system 50 and/or navigation system 70 and sends,transfers, copies and/or provides the population of images 402, 404taken during the first time interval associated with patient 10 to imageanalysis system 50 and/or navigation system 70. As shown in FIG. 5C,image analysis system 50 and/or navigation system 70 compiles thepopulation of images 402 at inspiration into a 3D image data subset 406of the anatomy of patient 10 and markers 22 at inspiration (referred toherein as inspiration 3D image data subset 406). Additionally, imageanalysis system 50 and/or navigation system 70 compiles the populationof images 404 at expiration into a 3D image data subset 408 of theanatomy of patient 10 at expiration (referred to herein as expiration 3Dimage data subset 408). The inspiration 3D image data subset 406 and theexpiration 3D image data subset 408 are then stored in an image dataset400 in memory component 54, 74 of image analysis system 50 and/ornavigation system 70.

Additionally, acquiring a population of images at both inspiration andexpiration may assist navigation of a steerable catheter during a secondtime interval. Referring now to FIGS. 6A-6C, in addition to segmentingthe markers 22 of PTD 20 from the population of images 402, 404generated during the first time interval, processor 52 of image analysisworkstation 50 generates three-dimensional models of the airway ofpatient 10 by segmenting the 3D image data subsets 406, 408. In variousembodiments, segmentation of the airway may be accomplished using aniterative region growing technique wherein a seed voxel in the airway isselected as an initialization parameter. Voxels neighboring the seedvoxel are then evaluated to determine whether they are a part of theairway, form the wall surrounding the airway, or form other tissue.Following segmentation, a surface mesh of the airway may be generated toproduce a surface skeleton. The surface of the airway may then berendered. Other segmentation methods are described herein below,including methods for localized segmentation extension in a region ofinterest.

As shown in FIG. 6A, a three-dimensional model of the airway of patient10 at inspiration (“inspiration 3D airway model 410”) is generated bysegmenting the inspiration 3D image data subset 406. FIG. 6A shows anInspiration/arms-up pathway registration; this is, generally speaking,the preferred image scan acquisition state for automatic segmentation ofthe tracheo-bronchial tree. Processor 52 may also segment one or moretarget tissues 420 (e.g., lesions, lymph nodes, blood vessels, tumors,etc.) which may be navigated to during a second time interval using avariety of medical devices as described more fully elsewhere herein. Thesegmentation of the target tissue(s) 420 may be refined to definedifferent characteristics of the target tissue, such as, for example,density of the target tissue. Additional image data formats may also beloaded into processor 52, such as, for example, PET or MR and processor52 may be able to map the CT, PET, and/or MR data to one another. Asdescribed herein below, an initial segmentation of an image(s) or imagedataset(s) can be extended in a region of interest to aid in navigationor for other use.

As shown at FIG. 6B, a three-dimensional model of the airway of patient10 at expiration (“expiration 3D airway model 412”) is generated bysegmenting the expiration 3D image data subset 408. As discussed above,a variety of segmentation algorithms known in the art may be used togenerate the initial inspiration and expiration 3D airway models 410,412. Methods for extending the initial segmentation are described hereinbelow. FIG. 6B shows, in contrast to FIG. 6A, an FRC/arms-downsegmentation. Because the patient's 10 lungs are more full of air atinspiration than at expiration, the inspiration 3D airway model 410includes more structure than the expiration 3D airway model 412.Accordingly, as shown in FIG. 6B, expiration 3D airway model 412includes fewer structure(s) and the structure(s) are in differentlocations and/or orientations than at inspiration. However, during aprocedure such as directing a navigated steerable catheter to a targettissue within the airway of patient 10 (e.g., during a second timeinterval), the breathing cycle of patient 10 may be closer to tidalbreathing. That is, patient 10 usually never reaches full inspirationduring the procedure and thus if the segmentation of the airways ofpatient 10 at inspiration is used for navigation purposes, there will besignificant error in the registration of the segmented airway to patient10.

In certain embodiments, a hybrid “Inspiration-Expiration” 3D airwaymodel 414 is constructed as shown in FIG. 6C using the inspiration 3Dairway model 410 and the expiration 3D airway model 412. The hybrid“Inspiration-Expiration” 3D airway model 414 may be used to reduce oreliminate the errors in registration. To construct the hybrid“Inspiration-Expiration” 3D airway model 414, a population ofdeformation vector fields is calculated by processor 52, 72 of imageanalysis system 50 and/or navigation system 70. The deformation vectorfield comprises vectors from one or more voxels in the inspiration 3Dairway model 410 to one or more corresponding voxels in the expiration3D airway model 412. After the deformation vector field is calculated,the inspiration 3D airway model 410 is deformed to the expiration stateof patient 10 using the deformation vector field. Accordingly, thevoxels in the inspiration 3D airway model 410 are deformed to match thelocation, shape, and orientation of the airways of patient 10 atexpiration. This results in the hybrid “Inspiration-Expiration” 3Dairway model 414, wherein the hybrid “Inspiration-Expiration” 3D airwaymodel 414 contains all of the structural information of the airways ofpatient 10 depicted in inspiration 3D airway model 410. However, thisstructural information is now more closely matched to the location,shape, and orientation of the airways of patient 10 depicted inexpiration 3D airway model 412. Accordingly, the deformation vectorsrepresent not only a change in location of the structure of the airwaybut a change in shape of the structure of the airway from inspiration toexpiration.

FIG. 7 , illustrates a 3D representation of hybrid“Inspiration-Expiration” 3D airway model 414 which includes a targettissue 420 segmented by processor 52, 72. This 3D representation ofhybrid “Inspiration-Expiration” 3D airway model 414 may include surfaceinformation. Hybrid “Inspiration-Expiration” 3D airway model 414 mayadditionally include navigation pathway 416. Image analysis system 50and/or navigation system 70 may calculate navigation pathway 416 fromthe entry of the airway to the location of target tissue 420. In certainembodiments, navigation pathway 416 may be an optimal endobronchial pathto a target tissue. For example, navigation pathway 416 may representthe closest distance and/or closest angle to the target tissue. Aphysician or other healthcare professional may follow navigation pathway416 during an image guided intervention to reach the location of targettissue 420.

Although target tissue 420 locations and navigation pathway(s) 416 maybe automatically calculated by image analysis system 50 and/ornavigation system 70, a physician or other healthcare professional maymanually adjust target tissue 420 locations and/or navigation pathway(s)416. As described herein with reference to FIGS. 14-19 , the navigationpathway 416 can be improved based on additional information obtainedduring localized segmentation extension methods according to embodimentsset forth below.

In general, the embodiments described herein have applicability in“Inspiration to Expiration”-type CT scan fusion. According to variousmethods, the user navigates on the expiration 3D image data subset 408for optimal accuracy, while using the inspiration 3D image data subset406 to obtain maximum airway segmentation. In one embodiment, forexample, a user could complete planning and pathway segmentation on theinspiration 3D image data subset 406 of patient 10. Preferably, adeformation vector field is created between at least two datasets (e.g.,from inspiration 3D image data subset 406 to expiration 3D image datasubset 408). The deformation or vector field may then be applied to thesegmented vessels and/or airways and navigation pathway 416 and targettissue 420 locations. In these and other embodiments, the deformation orvector field may also be applied to multiple image datasets or in aprogressive way to create a moving underlying image dataset that matchesthe respiratory or cardiac motion of patient 10.

By way of example, in certain embodiments, “Inspiration to Expiration”CT fusion using the lung lobe centroid and vector change to modify anairway model may also be applicable. In accordance with variousembodiments, this technique is used to translate and scale each airwaybased on the lung lobe change between inspiration images and expirationimages. The lung is constructed of multiple lobes and these lobes arecommonly analyzed for volume, shape, and translation change. Each lobechanges in a very different way during the patient's breathing cycle.Using this information to scale and translate the airways that arelocated in each lobe, it is possible to adapt for airway movement. Thisscaled airway model may then be linked to the 4D tracking of the patientas described herein.

In various aspects, the systems and methods described herein involvemodifying inspiration images generated by imaging device 40 (e.g., CT,CT/PET, MRI, etc.) to the expiration cycle for navigation. It is wellunderstood that the patient's airways are contained within multiplelobes of the lung. It is also understood that airways significantlychange between inspiration and expiration. In certain embodiments, toincrease the accuracy of the map for navigation, it may be beneficial toinclude the detail of the inspiration images, coupled with the abilityto navigate it accurately during expiration. For many patients, theexpiration state may be the most repeatable point in a patient's breathcycle. In preferred embodiments, this modification may be carried out inaccordance with the following steps:

1) Generate a population of images of patient 10 at both inspiration andexpiration using imaging device 40;

2) Segment the airways in both the inspiration and expiration images;

3) Segment the lung lobes in both the inspiration and expiration images(as the lung lobes are identifiable in both the inspiration andexpiration images with a high degree of accuracy);

4) Determine a volume difference for each lung lobe between inspirationand expiration, use this change to shrink the airway size from theinspiration to the expiration cycle. Preferably, this is done for eachindividual lobe, as the percentage change will typically be differentfor each lobe.

5) Determine the centroid for each lung lobe and the vector change inmotion from the main carina in both inspiration images and expirationimages. This vector may then be used to shift the airways that areassociated with each lung lobe. A centroid for the airway may becalculated based on the segmented branches. For each airway branch inthe segmentation, it includes a tag that associates it with therespective lung lobe. The central airway including the main carina andinitial airway branches for each lobe that is linked according to theexpiration scan location of these points. Next, a plane may be definedusing the main carina and initial airway branch exits to determine thevector change for each lobe.

Among the lobes to modify, for example:

left inferior lobe—the bottom lobe of the lung on the left side ofpatient 10;

left superior lobe—the top lobe of the lung on the left side of patient10.

right inferior lobe—the bottom lobe of the lung on the right side ofpatient 10;

right middle lobe—the middle lobe of the lung on the right side ofpatient 10;

right superior lobe—the top lobe of the lung on the right side ofpatient 10.

Exemplary calculations are as follows:

Inspiration Airway—Left Inferior Lobe (LIL)×70% (reduction in volumeInspiration to Expiration calculated)=ExAirwayLIL;

Determine Expiration Central Airway points (Main Carina and InitialAirway branches) based upon segmentation;

Shift ExAirwayLIL by vector distance (3 cm, 45 degrees up and back frommain carina) that LIL centroid moved from inspiration to expiration.

Preferably, this process is repeated for each lobe. In certainembodiments, the completion of 5 lobes will result in a hybrid“Inspiration-Expiration” 3D airway model for patient 10.

In various embodiments, the target location for the patient may beselected in the expiration images and applied to the hybrid“Inspiration-Expiration” 3D airway model 414. Alternatively, it may beselected in the inspiration images and adjusted based on the same orsimilar criteria as the inspiration airways. In either case, it may beadjusted individually or linked to the airway via a 3D network and movedin the same transformation.

A deformation field may also be included in the analysis in variousother embodiments described herein. For example, the deformation fieldmay be applied to fuse 3D fluoroscopic images to CT images to compensatefor different patient orientations, patient position, respiration,deformation induced by the catheter or other instrument, and/or otherchanges or perturbations that occur due to therapy delivery or resectionor ablation of tissue.

Following the generation of hybrid “Inspiration-Expiration” 3D airwaymodel 414, during a second time interval, a medical procedure is thenperformed on patient 10 with PTD 20 coupled to patient 10 at the samelocation as during the first time interval when the population ofpre-procedural images were taken. Preferably, the second time intervalimmediately follows the first time interval. However, in certainembodiments, second time interval may occur several hours, days, weeksor months after the first time interval. After hybrid“Inspiration-Expiration” 3D airway model 414 is generated and one ormore target tissues 420 are identified and one or more navigationpathways 416 are calculated, this information is transferred from imageanalysis system 50 to navigation system 70. This transfer may be doneaccording to the DICOM (Digital Imaging and Communications in Medicine)standard as known in the art. It will be understood that the transfermay be done using any method and according to any standard withoutdeparting from the scope of the invention. For example, this transfermay be accomplished between image analysis system 50 to navigationsystem 70 using a variety of methods, including, but not limited to, awired connection, a wireless connection, via CD, via a USB device, viadisk, etc.

It should be noted that image dataset 400 may be supplemented, replacedor fused with an additional image dataset. In one embodiment, forexample, during the second time interval an additional population ofimages may be taken. In other embodiments, for example, after the secondtime interval an additional population of images may be taken. Bygenerating one or more additional image datasets, potential changedphysical parameters of patient such as patient 10 movement, anatomicalchanges due to resection, ablation, general anesthesia, pneumothorax,and/or other organ shift may be accounted for during the procedure.Accordingly, images from CT-Fluoro, fluoroscopic, ultrasound or 3Dfluoroscopy may be imported into image analysis system 50 and/ornavigation system 70.

Using the respiratory signal derived from PTD 20, navigation system 70selects an image from the population of pre-procedural images 402, 404taken during the first time interval that indicates a distance or isgrouped in a similar sequence of motion between corresponding markers 22at a given instant in time, that most closely approximates or matchesthe distance or similar sequence of motion between the selectedlocalization elements 24. The process of comparing the distances isdescribed in more detail below. Thus, navigation system 70 displaysimages corresponding to the actual movement of the targeted anatomyduring the medical procedure being performed during the second timeinterval. The images illustrate the orientation and shape of thetargeted anatomy during a path of motion of the anatomy, for example,during inhaling and exhaling.

FIG. 8 illustrates an example set of distances or vectors d1 through d6between a set of markers 22, labeled ml through m9 that are disposed atspaced locations on PTD 20. As described above, a population ofpre-procedural images is taken of a patient 10 to which PTD 20 iscoupled during a first time interval. The distances between markers 22are determined for multiple instants in time through the path of motionof the dynamic body (e.g., the respiratory cycle of the patient). Then,during a medical procedure, performed during a second time interval,localization elements 24 (not shown in FIG. 8 ) proximate the locationof markers 22 provide position data for localization elements 24 tolocalization device 76 (not shown in FIG. 8 ). Navigation system 70 usesthe position data to determine distances or vectors between localizationelements 24 for multiple instants in time during the medical procedureor second time interval.

FIG. 9A shows an example of distance or vector data from localizationdevice 76. Vectors al through a6 represent distance data for one instantin time and vectors n1 through n6 for another instant in time, during atime interval from a to n. As previously described, the vector data maybe used to select an image from the population of pre-procedural imagesthat includes distances between the markers ml through m9 thatcorrespond to or closely approximate the distances al through a6 fortime a, for example, between the localization elements. The same processmay be performed for the vectors n1 through n6 captured during time n.

One method of selecting the appropriate image from the population ofpre-procedural images 402, 404 is to execute an algorithm that sums allof the distances al through a6 and then search for and match this sum toan image containing a sum of all of the distances d1 through d6 obtainedpre-procedurally from the image data that is equal to the sum of thedistances al through a6. When the difference between these sums is equalto zero, the relative position and orientation of the anatomy or dynamicbody D during the medical procedure will substantially match theposition and orientation of the anatomy in the particular image. Theimage associated with distances d1 through d6 that match or closelyapproximate the distances al through a6 may then be selected anddisplayed. For example, FIG. 9B illustrates examples of pre-proceduralimages, Image a and Image n, of a dynamic body D that correspond to thedistances al through a6 and n1 through n6, respectively. An example ofan algorithm for determining a match is as follows:Does Σ ai=Σdi(i=1 to 6 in this example) ORDoes Σ(ai−di)=0(i=1 to 6 in this example).

If yes to either of these, then the image is a match to the vector ordistance data obtained during the medical procedure.

FIG. 10 is a flowchart illustrating a method according to an embodimentof the invention. A method 100 includes at step 102 generating imagedata during a pre-procedural or first time interval. As discussed above,a population of images are generated of a dynamic body, such as patient10, using imaging device 40 (e.g., CT Scan, MRI, etc.). The image datais associated with one or more images generated of PTD 20 coupled to adynamic body, where PTD 20 includes two or more markers 22. In otherwords, the image data of the dynamic body is correlated with image datarelated to PTD 20. The one or more images may be generated using avariety of different imaging devices as described previously. The imagedata include an indication of a position of a first marker and anindication of a position of a second marker, as illustrated at step 104.The image data include position data for multiple positions of themarkers during a range or path of motion of the dynamic body over aselected time interval. As described above, the image data includeposition data associated with multiple markers, however, only two aredescribed here for simplicity. A distance between the position of thefirst marker and the position of the second marker is determined formultiple instants in time during the first time interval, at step 106.As also described above, the determination may include determining thedistance based on the observable distance between the markers on a givenimage. The image data, including all of the images received during thefirst time interval, the position, and the distance data is recorded ina memory component at step 108.

Then at step 110, during a second time interval, while performing amedical procedure on patient 10 with PTD 20 positioned on patient 10 atsubstantially the same location, position data is received for a firstlocalization element and a second localization element. Localizationelements 24 of PTD 20 are proximate markers 22, such that the positiondata associated with localization elements 24 is used to determine therelative position of markers 22 in real-time during the medicalprocedure. The position data of localization elements 24 are recorded ina memory component at step 112.

A distance between the first and second localization elements isdetermined at step 114. Although only two localization elements 24 aredescribed, as with the markers, position data associated with more thantwo localization elements may be received and the distances between theadditional localization elements may be determined.

The next step is to determine which image from the population of imagestaken during the first time interval represents the relative positionand/or orientation of the dynamic body at a given instant in time duringthe second time interval or during the medical procedure. To determinethis, at step 116, the distance between the positions of the first andsecond localization elements at a given instant in time during thesecond time interval determined in step 114 are compared to thedistance(s) determined in step 106 between the positions of the firstand second markers obtained with the image data during the first timeinterval.

An image is selected from the first time interval that best representsthe same position and orientation of the dynamic body at a given instantin time during the medical procedure. To do this, the difference betweenthe distance between a given pair of localization elements during thesecond time interval is used to select the image that contains the samedistance between the same given pair of markers from the image datareceived during the first time interval. This is accomplished, forexample, by executing an algorithm to perform the calculations. Whenthere are multiple pairs of markers and localization elements, thealgorithm may sum the distances between all of the selected pairs ofelements for a given instant in time during the second time interval andsum the distances between all of the associated selected pairs ofmarkers for each instant in time during the first time interval when thepre-procedural image data was received.

When an image is found that provides the sum of distances for theselected pairs of markers that is substantially the same as the sum ofthe distances between the localization elements during the second timeinterval, then that image is selected at step 118. The selected image isthen displayed at step 120. The physician or other healthcareprofessional may then observe the image during the medical procedure.Thus, during the medical procedure, the above process may becontinuously executed such that multiple images are displayed and imagescorresponding to real-time positions of the dynamic body may be viewed.

In addition to tracking the location of PTD 20, navigation system 70(see FIG. 3 ) may also track any type of device which includes one ormore localization elements. The localization elements in the medicaldevices may be substantially similar or identical to localizationelements 24 of PTD 20. The devices preferably include medical devices,including, but not limited to, steerable catheters, needles, stents,ablation probes, biopsy devices, guide wires, forceps devices, brushes,stylets, pointer probes, radioactive seeds, implants, endoscopes, energydelivery devices, therapy delivery devices, delivery of energy activatedsubstances (e.g., porfimer sodium) and energy devices, radiofrequency(RF) energy devices, cryotherapy devices, laser devices, microwavedevices, diffuse infrared laser devices, etc. In certain embodiments,the location of these devices is tracked in relation to PTD 20. In otherembodiments, for example, these devices are tracked in relation toelectromagnetic field generator 62, 82. It is also envisioned that atleast some of these medical devices may be wireless or have wirelesscommunications links. It is also envisioned that the medical devices mayencompass medical devices which are used for exploratory purposes,testing purposes or other types of medical procedures. Tracked ornavigated devices can be used to supplement segmentation information, asdescribed herein below.

One embodiment of a medical device which may be tracked by navigationsystem 70 is illustrated in FIGS. 11 and 11A. In one embodiment of thepresent invention, a navigated surgical catheter that is steerable 600(referred herein to as “steerable catheter”) may be used to gain accessto, manipulate, remove, sample or otherwise treat tissue within the bodyincluding, but not limited to, for example, heart or lung tissue.Steerable catheter 600 comprises an elongate flexible shaft 602 having aproximal end portion 604, a distal end portion 606 terminating in tip607, and one or more working channels 608 extending from proximal endportion 604 to tip 607. As shown in FIG. 11A, one or more localizationelements 610 that are detectable by navigation system 70 are disposedproximate the distal end portion 606 of elongate flexible shaft 602.Accordingly, the position and orientation (POSE) of localizationelements 610 are tracked by localization device 76 of navigation system70. The one or more localization elements 610 are connected by wire 611to navigation system 70; in alternative embodiments, the one or morelocalization elements 610 may be wirelessly connected to navigationsystem 70. In certain embodiments, localization elements 610 comprisesix (6) degree of freedom (6DOF) electromagnetic coil sensors. In otherembodiments, localization elements 610 comprise five (5) degree offreedom (5DOF) electromagnetic coil sensors. In other embodiments,localization elements 610 comprise other localization devices such asradiopaque markers that are visible via fluoroscopic imaging andechogenic patterns that are visible via ultrasonic imaging. In yet otherembodiments, localization elements 610 may be, for example, infraredlight emitting diodes, and/or optical passive reflective markers.Localization elements 610 may also be, or be integrated with, one ormore fiber optic localization (FDL) devices. Accordingly, in certainembodiments, localization elements 610 may be substantially similar oridentical to localization elements 24 of PTD 20. In other embodimentsthe steerable catheter may be non-navigated, such that it does notinclude any localization elements.

Steerable catheter 600 further comprises handle 612 attached to theproximal end portion 604 of elongate flexible shaft 602. Handle 612 ofsteerable catheter 600 includes steering actuator 614 wherein distal endportion 606 is moved “up” and “down” relative to proximal end portion604 by manipulating steering actuator 614 “up” and “down,” respectively.Additionally, distal end portion 606 is moved “left” and “right”relative to proximal end portion 604 by rotating handle 612 “left” and“right,” respectively, about handle longitudinal axis 613. It will beunderstood that steering actuator 614 and handle 612 are connected to asteering mechanism (not shown) on the inside of steerable catheter 600which is connected to distal end portion 606 of elongate flexible shaft602 for causing the deflection in distal end portion 606. Port 616,disposed on handle 612, provides access to working channel(s) 608 inelongate flexible shaft 602 of steerable catheter 600, such that amedical device may be inserted into working channel(s) 608 through port616.

As shown in FIGS. 12A and 12B, any number of medical devices ortherapies may be inserted into working channel(s) 608 and/or extendedout of tip 607 to deliver the medical devices or therapies to a targettissue. The medical devices may include, but are not limited to, imagingdevices 633, tissue sensing devices 632, biopsy devices, therapydevices, steerable catheters, endoscopes, bronchoscopes, percutaneousdevices, percutaneous needles, pointer probes, implants, stents, guidewires, stylets, etc. In certain embodiments, imaging devices 633include, but are not limited to, bronchoscopic video cameras 630,endobronchial ultrasound (EBUS) devices 634, optical coherencetomography (OCT) devices, probe based Confocal Laser Endomicroscopy(pCLE) devices, or any known imaging device insertable into workingchannel 608 of steerable catheter 600. Tissue sensing device 632 may beany type of device which may be used to determine the presence of atarget tissue in patient 10. In certain embodiments, tissue sensingdevice 632 may include, but is not limited to, imaging device 633, acell analysis device, a cancer detecting device, an exhaled breathcondensate analyzer, a physiological characteristic sensor, a chemicalanalysis device, an aromatic hydrocarbon detection device, vacuumcollection device, etc. The sensitivity of certain of the tissuesampling devices, such as aromatic hydrocarbon detection devices aredependent upon the density of the sample collected. Thus, by navigatingsteerable catheter 600 near the desired target tissue a sample of higherdensity may be captured and analyzed. Additionally, a vacuum collectiondevice may be navigated using steerable catheter 600 to near the desiredtarget tissue and/or an airway branch within one or two segments of thedesired target tissue, and an air sample may be captured. In certainembodiments, therapy devices include, but are not limited to, ablationprobes, energy delivery devices, radioactive seeds, delivery of energyactivated substances (e.g., porfimer sodium) and energy devices,radiofrequency (RF) energy devices, cryotherapy devices, laser devices,microwave devices, diffuse infrared laser devices, fluids, drugs,combinations thereof, or the like). In certain embodiments, biopsydevices include, but are not limited to, needles, forceps devices,brushes, etc. In certain embodiments, steerable catheter 600 may alsoinclude a suction capability.

As illustrated in FIG. 12A, for example, in certain embodiments, imagingdevice 633 is a bronchoscopic video camera 630. Bronchoscopic videocamera 630 may be inserted into working channel 608 and/or extended outdistal end portion 606 of navigated steerable catheter 600. By insertingbronchoscopic video camera 630 into working channel 608 of steerablecatheter 600, steerable catheter 600 may be used like a typicalsteerable bronchoscope, as described more fully elsewhere herein.

As shown in FIG. 12B, tissue sensing device 632 may be an imaging device633, wherein imaging device 633 is an endobronchial ultrasound (EBUS)device 634; however, as described above, it will be understood thatimaging device 633 may include, but is not limited to, bronchoscopicvideo camera 630, an optical coherence tomography (OCT) device, a probebased Confocal Laser Endomicroscopy (pCLE) device, or any known imagingdevice insertable into working channel 608 of steerable catheter 600.

In embodiments, where tissue sensing device 632 is imaging device 633,imaging device 633 may be able to generate a population of images of thetarget tissue(s), wherein the target tissue(s) may be in the airway, onthe wall of the airway, in the wall of the airway, and/or beyond thewall of the airway. That is, the imaging device(s) may be able togenerate images of target tissue(s) not only inside the airway but maygenerate images of target tissue(s) and other anatomical structuresinside and/or beyond the wall of the airway. Thus, in certainembodiments, sub-surface target tissue may be imaged using the imagingdevice(s). Accordingly, using endobronchial ultrasound (EBUS) device634, an optical coherence tomography (OCT) device, a probe basedConfocal Laser Endomicroscopy (pCLE) device, etc. while tracking theposition and orientation (POSE) of localization element 610 of steerablecatheter 600, as described herein, multiple 3D volumes of image dataregarding target tissue(s) and other anatomical structures inside and/orbeyond the wall of the airway may be collected and a larger 3D volume ofcollected data may be constructed. Knowing the 3D location andorientation of the multiple 3D volumes will allow the physician or otherhealthcare professional to view a more robust image of, for example,pre-cancerous changes of target tissue(s) in patient 10. This data mayalso be correlated to pre-acquired or intra-procedurally acquired imagedataset 400 to provide additional information.

Additionally, in certain embodiments wherein steerable catheter 600includes multiple working channels 608, multiple medical devices may beinserted into the multiple working channels 608. For example,bronchoscopic video camera 630 may be inserted into one working channeland a medical device such as a needle, forceps device or a brush may beinserted into a second working channel. Accordingly, a real-time imagefeed from bronchoscopic video camera 630 may be used to view theoperation of the medical device. Although a steerable catheter has beendescribed, it will be understood that any type of steerable medicaldevice may be used in accordance with the methods described herein,including, but not limited to, endoscopes, bronchoscopes, etc. withoutdeparting from the scope of the invention. It is understood that othermedical devices may be tracked by navigation system 70, such as apercutaneous needle or other device, such as described in U.S. Ser. No.15/290,822, for example, the entirety of which is incorporated herein byreference.

In various embodiments, any of the medical devices described herein thatmay be inserted into working channel(s) 608, 658 of steerable catheter600 and/or other medical devices may be tracked individually with anintegrated localization element (e.g., an electromagnetic (EM) sensor).Accordingly, the medical devices may be tip tracked. Additionally,wherein the inserted medical device is an ablation probe, ablationmodels may be displayed to assist in optimal placement of the ablationprobe for treatment. It will be understood that the medical devices maybe delivered endobronchially, percutaneously, and/or endobronchially andpercutaneously simultaneously.

Referring again to navigation system 70, navigation system 70 maydisplay on display 80 multiple images which may assist a physician orother healthcare professional in conducting the methods describedherein. Image dataset 400 generated during the first time interval maybe registered to patient 10 using PTD 20. As described above,localization elements 24 of PTD 20 are proximate markers 22 and becauseone or more markers 22 of PTD 20 are visible in image dataset 400 andlocalization elements 24 corresponding to the one or more markers 22 aretracked by navigation system 70, image dataset 400 may be registered topatient 10. This registration may be manually accomplished or may beautomatically accomplished by navigation system 70.

In addition to or alternative to registration using PTD 20, registrationmay be completed by different known techniques. First, point-to-pointregistration may be accomplished by identifying points in an image spaceand then touching the same points in patient space. These points aregenerally anatomical landmarks that are easily identifiable on thepatient. Second, lumen registration may be accomplished by generating apoint cloud within the airways of patient 10 and matching the shape ofthe generated point cloud to an inspiration 3D airway model 410, anexpiration 3D airway model 412, and/or a hybrid “Inspiration-Expiration”3D airway model 414. Using four-dimensional tracking (4D) the pointcloud may be generated at the appropriate respiration cycle to matchinspiration 3D airway model 410, an expiration 3D airway model 412,and/or a hybrid “Inspiration-Expiration” 3D airway model 414. Generationof a point cloud is more fully described in U.S. Ser. No. 13/773,984,entitled “Systems, Methods and Devices for Forming Respiratory-GatedPoint Cloud for Four Dimensional Soft Tissue Navigation,” filed on Feb.22, 2013, which is hereby incorporated by reference. Third, surfaceregistration may involve the generation of a surface in patient 10 spaceby either selecting multiple points or scanning, and then accepting thebest fit to that surface in image space by iteratively calculating withprocessor 72 until a surface match is identified. Fourth, repeatfixation devices entail repeatedly removing and replacing a device(i.e., dynamic reference frame, etc.) in known relation to patient 10 orimage fiducials of patient 10. Fifth, two-dimensional (2D) imagedatasets may be registered to three-dimensional (3D) image datasetswherein, the two dimensional image datasets may include, but are notlimited to, fluoroscopic images, ultrasound images, etc. and thethree-dimensional (3D) image datasets may include, but are not limited,to computed tomography (CT) images, fused computed tomography-positronemission tomography (CT/PET) images, magnetic resonance imaging (MRI)images. Sixth, automatic registration may be accomplished by firstattaching a dynamic reference frame to patient 10 prior to acquiringimage data. It is envisioned that other known registration proceduresare also within the scope of the present invention, such as thatdisclosed in U.S. Pat. No. 6,470,207, entitled Navigational Guidance viaComputer-Assisted Fluoroscopic Imaging”, filed on Mar. 23, 1999, whichis hereby incorporated by reference.

After image dataset 400 is registered to patient 10, navigation system70 displays on display 80 a variety of images as illustrated in FIG. 13. For example, as shown in panel 700, hybrid “Inspiration-Expiration” 3Dairway model 414 may be displayed. Additionally, as shown in panel 700,an indicia 718 (shown as a crosshair) of the location of steerablecatheter 600 is displayed. In certain embodiments, for example, indicia718 indicates the location of distal end portion 606 of steerablecatheter 600. In other embodiments, for example, indicia 718 indicatesthe location of localization element 610 of steerable catheter 600. Inyet other embodiments, for example, indicia 718 indicates the locationof tip 607 of steerable catheter 600. That is, navigation system 70 maybe able to display an indicia indicating the location of a portion ofsteerable catheter 600 based on the tracked location of localizationelement 610. For example, if localization element 610 is disposed 5 mmfrom tip 607 of steerable catheter 600, the 5 mm distance may be takeninto account by navigation system 70 and the indicia of tip 607indicating the location of tip 607 may be displayed and not the locationof localization element 610. An indicia 720 (shown as a circle) of aninitial target tissue location may also be displayed on display 80 bynavigation system 70 as shown in panel 700. Indicia 718, 720 are shownas a crosshair and circle, respectively; however, it is envisioned thatother indicia may be used to indicate the location of steerable catheter600, initial target tissue location, confirmed target tissue location,location of a percutaneous needle, and/or any other target tissue ormedical device. For example, indicia may have different shapes, colors,sizes, line weights and/or styles, etc. without departing from the scopeof the invention.

Furthermore, navigation system 70 may be able to simulate and displayaxial, coronal and oblique images based on the position and orientation(POSE) of localization element 610 of steerable catheter 600, as shownin panels 702, 704, and 706. To simulate these views, navigation system70 may modify one or more images from image dataset 400 using knownimage manipulation techniques. Additionally, navigation system 70 maysimulate and/or display orthogonal image slices, oblique or off-axisimage slices, volume rendered images, segmented images, fused modalityimages, maximum intensity projection (MIPS) images, video, and videoenhanced images. As shown, indicia of 718 of steerable catheter 600and/or an indicia 720 of an initial target tissue location may also bedisplayed, as shown in panels 702, 704, and 706.

In various embodiments as shown in panel 712, navigation system 70 alsosimulates a virtual volumetric scene within the body of patient 10, suchas the airways of patient 10, from a point of view of a medical device,such as steerable catheter 600, as it is being navigated into and/orthrough patient 10. This virtual volumetric scene is acomputer-generated visualization of a bronchoscopy procedure andsimulates what would be viewed by a bronchoscopic video camera insertedinto the airways. To simulate the virtual volumetric scene, navigationsystem 70 modifies one or more images from image dataset 400 using knownimage manipulation techniques. For example, navigation system 70 may beable to simulate the virtual volumetric scene using inspiration 3Dairway model 410, expiration 3D airway model 412, and/or hybrid“Inspiration-Expiration” 3D airway model 414. Accordingly, navigationsystem 70 renders an internal view of 3D airway model(s) 410, 412,and/or 414 based on a virtual bronchoscope video camera position, forexample, by applying certain surface properties (e.g., Lambertian),diffuse shading model(s), and perspective projection camera model(s).Virtual lighting and shading may be applied to the rendered view tofurther enhance the virtual volumetric scene. The field of view (FOV)may be changed to match the field of view of bronchoscopic video camera630 (see FIG. 12A). The point of view may be adjusted to matchbronchoscopic video camera 630 or to display a virtual volumetric scenefrom different points along the airway or outside the airway. Navigationsystem 70 may also be able to display a navigation pathway 416 in thevirtual volumetric scene. Accordingly, the virtual volumetric scene mayallow a physician or other healthcare professional to review thenavigation pathway 416 prior to inserting steerable catheter 600 and/orother medical device into patient 10. Additionally, in certainembodiments, an indicia of the location of localization element 610 ofsteerable catheter 600 and/or an indicia of an initial target tissuelocation may also be displayed.

Additionally, in various embodiments as shown in panel 716, navigationsystem 70 also displays a real-time image feed from bronchoscopic videocamera 630 inserted into working channel 608 of steerable catheter 600.The real-time image feed may be static images or moving video. Thereal-time image feed may assist the physician or other healthcareprofessional in navigating steerable catheter 600 to proximate theinitial location of the target tissue. Thus, by inserting bronchoscopicvideo camera 630 into working channel 608 of steerable catheter 600 (seeFIG. 12A), steerable catheter 600 may be used like a typical steerablebronchoscope. Typical steerable bronchoscopes are used to visuallyinspect the airways of a patient and have a fixed bronchoscopic videocamera in addition to one or more working channels. Typical steerablebronchoscopes may have steering actuators and steering mechanisms thatpermit them to be steered much like steerable catheter 600. Because thebronchoscopic video camera of a typical steerable bronchoscope is fixedduring manufacture of the steerable bronchoscope, the “up” orientationof the image feed from the bronchoscopic video camera as displayed tothe physician or other healthcare professional is aligned with the “up”direction of the steering actuator of the typical steerablebronchoscope. However, it may be desirable to use steerable catheter 600which may have a smaller outside diameter than the typical steerablebronchoscope. The image feed from bronchoscopic video camera 630inserted into the working channel of the steerable catheter 600 can beoriented and aligned with the physician's view, as shown and describedin U.S. Ser. No. 15/290,822, the entirety of which is incorporatedherein by reference. As further described in U.S. Ser. No. 15/290,822,the real-time image feed from the bronchoscopic video camera 630 can beregistered to the steerable catheter 600 and displayed by the navigationsystem 70.

Returning to FIG. 13 , navigation system 70 may also display a graphicalrepresentation 708 of the respiratory cycle of patient 10 monitoredusing PTD 20. In certain embodiments, one or more of the images and/orindicia displayed in panels 700, 702, 704, 706, 712 and 716 aredisplayed as a function of the monitored respiratory state. That is,images in image dataset 400 and/or generated from image dataset 400 aredisplayed on display 80 that depict the anatomy of patient 10 at themonitored respiratory state. For example, when the patient is atexpiration as monitored by PTD 20, images of the anatomy of the patientdepicting the anatomy at expiration are displayed. Accordingly, when thepatient is at inspiration as monitored by PTD 20, images of the anatomyof patient 10 depicting the anatomy at inspiration are displayed. Inother embodiments, one or more of the images displayed in panels 700,702, 704, 706, 712 and 716 may not be displayed as a function of themonitored respiratory state. That is, images in image dataset 400 and/orgenerated from image dataset 400 are displayed on display 80 that depictthe anatomy of patient 10 at one respiratory state. For example, whenthe patient is at expiration and inspiration as monitored by PTD 20,images of the anatomy of patient 10 depicting the anatomy at expirationare displayed. In embodiments where images are not displayed accordingto the monitored respiratory state, an indication 710 of whether thedisplayed images match the monitored respiratory state may be shown(e.g., “Respiration Matched”, “Respiration out of Sync”).

Additionally, the display of indicia of the locations of the targettissue and/or indicia of the location of various medical devices may besynchronized or gated with an anatomical function, such as the cardiacor respiratory cycle, of patient 10. That is, in certain embodiments,the indicia are displayed on display 80 as a function of the monitoredrespiratory state. In certain instances, the cardiac or respiratorycycle of patient 10 may cause the indicia to flutter or jitter withinpatient 10. In these instances, the indicia will likewise flutter orjitter on the image(s) displayed on display 80.

To eliminate the flutter of the indicia on the displayed image(s), theposition and orientation (POSE) of localization elements 610, 660 isacquired at a repetitive point within each cycle of either the cardiaccycle or the respiratory cycle of patient 10. To synchronize theacquisition of position data for localization elements 610, 660,navigation system 70 may use a timing signal (e.g., respiratory phasesignal) generated by PTD 20; however one skilled in the art will readilyrecognize other techniques for deriving a timing signal that correlateto at least one of the cardiac or respiratory cycle or other anatomicalcycle of the patient.

As described above, the indicia indicate the location of steerablecatheter 600 based on the location of localization element 610 trackedby navigation system 70 as steerable catheter is navigated by thephysician or other healthcare profession on and/or within patient 10.Rather than display the indicia on a real-time basis, the display of theindicia may be periodically updated based on the timing signal from PTD20. In various embodiments, PTD 20 may be connected to navigation system70. Navigation system 70 may then track localization elements 610, 660in response to a timing signal received from PTD 20. The position of theindicia may then be updated on display 80. It is readily understood thatother techniques for synchronizing the display of an indicia based onthe timing signal are within the scope of the present invention, therebyeliminating any flutter or jitter which may appear on the displayedimage(s). It is also envisioned that a path (or projected path) ofsteerable catheter 600, a percutaneous needle, and/or other medicaldevices may also be illustrated on the displayed image(s). The locationof a target in the lung of a patient can be confirmed endobronchiallyutilizing the devices, systems, and/or methods described herein and asset forth in U.S. Ser. No. 15/290,822, the entirety of which isincorporated herein by reference.

As described above, the processor 52 of the image analysis system 50and/or the processor 72 of the navigation system 70 utilize imageprocessing and segmentation techniques to identify components in animage, multiple images, or one or more image datasets (e.g., imagedatasets from different time periods). The segmentations are used duringthe navigation methods as described above.

Existing methods for segmentation of anatomical treelike structures(e.g., airways, pulmonary vasculature), vary in degree of accuracy andprocessing time. Spillage occurs when voxels in an image are denoted aspart of the segmentation which should not be included. To avoidspillage, many segmentation methods conservatively assess portions ofthe segmentation where spillage likely occurred, especially severalgenerations deep in the treelike structure, and err on the side ofexcluding those portions. As a result, many high generation branches onthe periphery of the treelike structure fail to be included in theresulting segmentation. However, for many segmentation uses (e.g., inimage-guided pulmonary biopsies or other procedures), it can beclinically detrimental to be missing high generation branches on theperiphery of the treelike structure (e.g., airway tree) near a targetarea. The best route to navigate to the target area may include pathwaysalong branches not included in the segmentation. If these highgeneration branches on the periphery of the treelike structure are notincluded in the segmentation, navigation systems cannot utilize thosebranches to plan the optimal path to the target area. Extendedsegmentation that includes as many high generation branches as possiblewill provide improved clinical yields and more effective procedures andnavigation to target areas.

Referring to FIGS. 15-19 , segmentation extension methods are shown anddescribed. The segmentation extension methods according to the presentdisclosure can be used in conjunction with the image analysis system 50,the navigation system 70, and/or the tracked medical devices (e.g.,steerable catheter 600) described above to enhance the segmentation ofthe image(s) or image dataset(s) (e.g., to increase the accuracy andoptimize the navigated pathway 416 to reach target tissue 420, forexample).

FIG. 14 illustrates a local extension method for segmentation ofanatomical treelike structures according to an embodiment. First, aninitial segmentation of the treelike structure is extracted from 3Dimage data (e.g., image(s), image dataset(s), etc.). At step 1100, 3Dimage data is received, for example loaded into a memory component froma disk or directly from an imaging device (e.g., imaging device 40 ofthe system described above). The 3D image data includes specifiedreference coordinates system (RCS) coordinates. In one embodiment, the3D image data includes a high resolution CT scan. At step 1102, the 3Dimage data is resampled to the RCS identity direction. Step 1102 isoptional to simplify subsequent steps of the method, and it isunderstood that resampling the image may be omitted within the scope ofthe present invention. For example, other steps in the disclosed methodcould account for non-identity directions if the image is not resampledto correspond to the identity matrix. At step 1104, the 3D image issegmented to obtain an initial segmentation of the anatomical treelikestructure (e.g., airway tree, pulmonary vascular tree, etc.). Step 1104can be performed using various segmentation techniques, such asHounsfield unit thresholding, convolution, connected component, or othersegmentation technique. For example, in one embodiment the initialsegmentation is performed using a custom airway segmentation library orusing a coarse segmentation that extracts the main branches of thetreelike structure and the first few generations of branches.

After the initial segmentation is extracted, at step 1106 a centerlineskeleton is created from the initial segmentation. The centerlineskeleton is topologically treelike (i.e., n bifurcation points with n−1centerlines connecting the bifurcation points). The centerline skeletonmay include a number of “branches” (or edges) corresponding to anordered list of 3D points (center points) that comprise a simple pathwith no loops or bifurcations. For example, a branch or edge may includeseveral center points that are connected to form the edge. At step 1108,each voxel in the initial segmentation is assigned to an edge in thecenterline skeleton (i.e., mask labeling each voxel by the “tree”edges). In this mask labeling step, each voxel is assigned to an edgesuch that the Euclidean distance between the corresponding geometricpoint of the voxel and the minimum distance to the centerline of theedge is minimized. Each edge will correspond to a generally cylindricaland convex mask containing the center points of the edge.

Referring still to FIG. 14 , at step 1110, a target area or point (forexample, a nodule on a lung, or target tissue 420 as described above)within the image space is defined. The target point is preferably withinreasonable proximity to the treelike structure. The target point can bedefined manually or automatically. In one embodiment, a user can selectthe target point using a graphical user interface, such as I/O component78 of the navigation system 70. In another embodiment, the target pointcan be automatically defined based on segmentation techniques (e.g.,Hounsfield unit thresholding). The defined target point identifies thelocation in image space for the segmentation extension to reach. In oneembodiment, the defined target point is a nodule (e.g. a centroid of asegmented nodule from the initial segmentation). In this embodiment, auser may desire additional treelike structure information extending tothe nodule, for example for navigation to the nodule for biopsy ortreatment. In another embodiment, the target area or point is a volumeof branch sparsity. In this embodiment, a user may desire to generallyimprove the segmentation of the treelike structure in an area where fewor no branches were identified in the initial segmentation, withoutnecessarily having a direct clinical application or need to reach thetarget point for a procedure. In other embodiments, the target point orarea can be determined by navigational information, such as informationfrom navigation system 70 obtained during a procedure, as described indetail elsewhere herein. In these embodiments, the navigationalinformation is used to define and extract the region of interest asdescribed below with reference to step 1116, without necessarilyidentifying an area or point for the extended segmentation to reach.

At step 1116, a region of interest for segmentation extension isextracted from the image. The region of interest is defined by abounding box based on the location of the target point and/or nearbybranches of the treelike structure. In one embodiment, a fixed-sizebounding box (e.g., 1 cm cube; 2 cm cube; 3 cm cube; any suitable fixedsize) is centered at the target point to define the region of interest.In a clinical application it is likely less helpful for the target pointto be centered in the region of interest, as branches beyond the targetpoint are not helpful for navigating to the target point. In anotherembodiment, the bounding box can be sized and positioned to encompassall center points within a given distance (e.g., 1 cm, 2 cm, 3 cm, anysuitable distance) of the target point. In another embodiment, a hybridapproach is used to define the bounding box, combining both the locationof the target point and the location of nearby branches. For example, abounding box can be defined to encompass all center points within agiven distance of the target point, and then extended towards the targetpoint. In one embodiment, illustrated in FIG. 15 , a hybrid approach todefining a region of interest includes searching for at most max and atleast min terminal branches within a given distance of the target point(step 1118). At step 1120, a percentage of the distal-most center pointsfrom the terminal branches are extracted. For example, in oneembodiment, at step 1120 the distal 30% of center points from theterminal branches are extracted. At step 1122, a bounding boxencompassing the extracted points from step 1122 is computed, and atstep 1124 a weighted average of the target point and corners of thebounding box is taken based on where the target point is in relation tothe box. The faces of the bounding box are extended toward the targetpoint as necessary based on the weighted average.

Referring again to FIG. 14 , at step 1130, the sub-image (i.e., theextracted region of interest) is smoothed. Smoothing the sub-image canhelp improve the accuracy of subsequent steps (e.g., tubular structurepoint detection). However, too much smoothing can have a detrimentaleffect in that points within the desired treelike structures will not bedetected. In one embodiment, bilateral smoothing can be used to smooththe sub-image. Bilateral smoothing includes performing Gaussiansmoothing while maintaining edges of the structures in the sub-image,which is beneficial for airway segmentations because it smooths withoutlosing the details of lumen walls. Alternatively, ordinary Gaussiansmoothing can be used. This smoothing step is optional and may beomitted within the scope of the present invention.

At step 1132, the contrast of the sub-image is enhanced. Enhancing thecontrast of the sub-image amplifies the first and second derivatives ofthe sub-image, which improves the accuracy of and reduces the processingtime for subsequent steps. The derivatives of the sub-image can beapproximated at a voxel using a value (e.g. Hounsfield unit) at thevoxel and the values at voxels in a neighborhood of the voxel.Amplifying contrast leaves the derivative values unaffected in regionsof the image where the voxel values are uniform and accentuates thederivative values in regions with varying voxel values. In oneembodiment, adaptive histogram equalization is used to enhance thecontrast, although any contrast enhancement can be used. This enhancingcontrast step is optional and may be omitted within the scope of thepresent invention.

At step 1134, voxels corresponding to tubular structures in thesub-image are detected. For example, voxels corresponding to tubularstructures in the sub-image having voxel values sharing qualities (e.g.,trajectory or range of trajectory, Hounsfield density within a certainrange (e.g., lower for lumens and higher for vasculature), etc.) withvoxels in the initial segmentation are detected. In this step, objectsof certain dimensionality (e.g., 1-dimensional line/tube structures) areextracted from the image based on a multi-scale Hessian analysis of thesub-image. The eigenvalues and eigenvectors of the Hessian analysis,computed discretely at different scales, are computed for each voxel.This information is used to assign each voxel a metric value based onhow tubular the voxel is. In one embodiment, the metric value is derivedfrom a weighted function incorporating the eigenvalues of the Hessiananalysis. In another embodiment, the span of the eigenvectorscorresponding to the two larger magnitude eigenvectors of the Hessiananalysis (the tubular structure's cross-section plane) is used to createa gradient vector flow vector field, which is then used to assign ametric value. Alternatively, a metric value can be obtained via a circlefitting method using the gradient vector flow vector field. The analysiscan be used to detect either relatively dark (e.g., airway lumens) orrelatively bright (e.g., vasculature) tubular structures. Once eachvoxel in the sub-image is assigned a metric value, thresholding is usedto extract only voxels with a “good” or desired metric value (i.e.,voxels satisfying a metric value criterion). For example, in oneembodiment, thresholding is used to extract only voxels having a metricvalue at least half the maximum metric value in the sub-image. Animaging processing library can be used to compute the Hessian imageanalysis and to evaluate the metric for each voxel.

Next, connected components are extracted from the thresholding result.For each component, the following information is extracted: the voxelwith the highest metric value, the physical point corresponding to thisvoxel, the voxel value at this voxel, the minimum and maximum voxelvalue within the component, the tube direction (which corresponds to theeigenvector for the smallest eigenvalue of the Hessian analysis at thevoxel), and the number of voxels in the component. In the case of airwaylumens, the tube direction is used to estimate the radius of the tube bycasting rays from the point in a circle that is perpendicular to thetube direction. The stopping condition for the ray is determined by achange in voxel value above a certain threshold. In one embodiment, thepercentage of “lumen wall hits” that occur before a maximum traversalstopping condition can also be recorded. For example, if intensityvalues along the ray increase beyond a certain threshold (e.g., 100 HU)given a certain step size (e.g., 0.2 mm), it is considered a “lumen wallhit.” It is expected that low intensity tubular structures encased byhigher intensity hollow tubes (lumen walls) will have a high percentageof such “lumen wall hits.” The information gathered for each componentis used to discard likely false positives and to pass information on tosubsequent steps of the extension method. Additional information can beextracted for each component. For example, in one embodiment, proximityto plate-like structures can be extracted for airway lumens. Theinformation that is extracted can be used as a feature vector, andmachine learning techniques can be employed to discard more falsepositives at this step 1134, although this is not required within thescope of the present invention.

As step 1136, the voxels detected in step 1134 are used as seed pointsfor spillage-constrained region growing. Each seed point that waslocated in step 1134 is used for region growing. In one embodiment, theregion growing method only grows voxels in a range that is the same asthose produced in the initial treelike segmentation. In one embodiment,the region growing method analyzes the growth after a certain number ofiterations to determine whether or not to stop based on the likelihoodof spillage. For example, the region growing method can include a cutofffor the maximum allowable percent growth, or other spillage-constrainingmechanism. In one embodiment, the region growing method includes ananalysis after a certain number of iterations to skeletonize and analyzethe shape of the growth (e.g., the elongation, the eccentricity, etc.)to determine the likelihood of spillage.

At step 1138, each resulting connected component based on the growth instep 1136 is extracted and labeled. The method can include thresholdingvalues to avoid spillage. For example, in one embodiment, componentswith less than a certain number of voxels (e.g., less than 20 voxels)are discarded.

Referring still to FIG. 14 , at step 1140 a centerline skeletonizationof each extracted component that is topologically treelike is created.This step is like step 1106 described above but performed on theextracted connected components from step 1138. At step 1142, each voxelin the extracted connected components is assigned to an edge in thecenterline skeleton (i.e., mask labeling each voxel by the “tree” edges,as described above with reference to step 1108).

At step 1144, the extracted connected components are pruned to removecomponents likely to be false positives. Entire components, segmentswithin components, or portions of segments within components can bediscarded based on image processing and/or geometric criteria in thispruning step. As shown in FIG. 16 , the pruning step 1144 includes theapplication of a variety of image processing and geometric constraintsto determine if a structure can be characterized as a sub-tree of theinitial treelike structure. The pruning method is based on certainassumptions about the characteristics of a valid component, segment, orportion (i.e., a component, segment, or portion that is a sub-tree ofthe initial treelike structure). It is assumed that valid components andtheir sub-segments should have genus zero (no holes) and be relativelyconvex, especially for segments. The pruning method analyzes thesecharacteristics based on a metric that takes the ratio of the componentor segment volume before morphological closing to its volume aftermorphological closing. Valid structures should score high on thismorphology metric. After morphologically closing, the volume will begreater than or equal to the volume before morphological closing, so theratio will be in the range of 0 to 1. The expected shapes for validstructures should have volumes that are roughly the same and relativelyinvariant before and after morphological closing (e.g., ratio of beforeto after morphological closing of about 0.925 or greater with astructuring radius of 1 for full components, or a dynamically definedratio with a larger structuring radius for segments of a component).Based on this assumption, at step 1146 the pruning includes discardingcomponents, segments, or portions based on the morphology metric. It isassumed that valid segments should not curve too much. The pruningmethod analyzes this characteristic by computing the dot product of thelocal segment direction (a small center point look-ahead), starting atits terminal end, and a more global direction (a larger look-ahead,e.g., to more center points). At step 1148, if this calculated curvingvalue is below a certain threshold, the segment is trimmed. It isassumed that valid segments should have appropriate numerical values forcertain shape analysis metrics, including minor axis length (derivedfrom the principal component analysis), elongation, and eccentricity. Inone embodiment, these shape analysis values can be taken from an imageprocessing library. At step 1150, components/segments/portions arediscarded if the shape analysis values are not appropriate. For example,in one embodiment a minor axis length of less than about 0.5 wouldtrigger pruning or discarding of the segment, component, or portion. Inone embodiment, an elongation of less than about 2.0 would triggerpruning or discarding of the segment, component, or portion. In oneembodiment, an eccentricity of less than about 0.8 would trigger pruningor discarding of the segment, component, or portion. It is understoodthat other shape analysis values can be applied in this pruning step,and that the condition metrics can be dynamic. It is assumed that forvalid segments, each contiguous sub-segment of at least a given lengthshould have about the same circle equivalent diameter (the diameter of acylinder which has the same length and volume as this segment, given thelength of its center point path). In one embodiment, a cylindrical scoreis based on the difference between the average of the three largestsub-segment circle equivalent diameters and the average of the threesmallest sub-segment circle equivalent diameters. Taking the averageaccounts for noise. At step 1152, components, segments, and/or portionsare discarded based on this cylindrical score. For example, in oneembodiment a cylindrical score metric (based on the difference betweenthe average of the greatest few and least few diameters as described)greater than about 1 mm would trigger pruning or discarding of thesegment, component, or portion. The pruning steps shown in FIG. 16 areperformed iteratively on a component until it is invariant for an entireiteration. In other words, the pruning steps are performed iterativelyuntil the component is not pruned or modified at all between consecutiveiterations. After each iteration, if the component is modified (e.g., ifsome curving is removed thereby modifying thecomponent/segment/portion), the skeletonization and mask labeling ofsteps 1140 and 1142 are recomputed for the component. In one embodiment,the characterization of valid subtrees and resultant pruning is enhancedby machine learning techniques, although this is optional. It isunderstood that additional pruning criteria can be employed to discardor modify potential components. For example, pre-defined shape modelscan be utilized to determine if potential components/segments/portionsfit expected or common airway structures. Bifurcation models (e.g., 2 or3 branch shapes) can be used to determine if the potential componentpoints fit high probability shapes for tree structures. The bifurcationmodels can be modified based on prior found airway branches toappropriate diameters, sizes, and configurations, based on locationwithin the tree structure.

At step 1160, the components that are determined to be valid after thepruning at step 1144 are connected back to the initial treelikestructure. The pruned components are connected back to the initialtreelike structure based on image processing and/or geometric criteria.Referring to FIG. 17 , at step 1162 it is determined whether thecomponent has boundary voxels that are 4-connected to the initialsegmentation. If so, voxels are filled in to fully connect the componentto the initial segmentation. If not, at step 1164 it is determinedwhether the component is geometrically compatible with the initialsegmentation. For each segment in the component, all terminal segmentsof the initial treelike structure within a certain distance, based onthe 3D Euclidean distance between center points, of the componentsegment are collected from the initial treelike structure. For example,in one embodiment all terminal segments of the initial treelikestructure having center points that are within about 2 cm to about 4 cm,based on the 3D Euclidean distance, of center points in the segment ofthe component are collected. For instance, in one embodiment terminalsegments of the initial treelike structure are collected starting at afirst distance (e.g., 2 cm) and continuing at greater distances within arange until a sufficient number of terminal segments are found. Theradius of each terminal segment is estimated at the terminal end, usinga technique similar to the one described above with reference to step1134. Then, a radius estimate for the current segment is calculated.If 1) the radii are compatible, 2) the terminal point of the terminalsegment is within a certain distance of the proximal point of thecomponent's segment, and 3) the dot product of the segment's directionsand the dot products of the directions with the vector from the distalend of the terminal segment to the proximal point of the component'ssegment are compatible (e.g., close to 1), then voxels are filled inconnecting the component to the initial segmentation. If both step 1162and step 1164 fail, but the segment of the component under considerationcontains a “path” of seed points determined using the points and Hessiandirections from step 1134, the probability of the component being validis assumed to be high. In this case, some of the constraints of step1164 are relaxed. If, upon relaxing the constraints of step 1164, theconstraints are compatible, a “gradient ball” method is employed at step1166 to determine new possible growth. First, starting at the componentsegment's proximal point, a ball with a radius determined by thesegment's estimated radius is centered at the point. Second, the voxelwith the maximum or minimum voxel value (e.g., based on whether thestructure is a vasculature or lumen structure) in the ball that is aboveor below a certain threshold and that falls within a cylinder of radiusdetermined by both the segments' radius estimates connecting the twosegments (component segment and terminal segment of initial treelikestructure) is found. For example, for lumen structures, the voxel withthe minimum voxel value in the ball that is below a threshold (e.g.,below −700 HU in one embodiment) and falls within a cylinder connectingthe component segment to the terminal segment of the initial treelikestructure is found. If there is no voxel below the threshold condition,this sub-method stops because there are no voxels satisfying thecondition. For vasculature structures, the voxel with the maximum voxelvalue in the ball that is above a threshold and falls within a cylinderconnecting the component segment to the terminal segment of the initialtreelike structure is found. If there is no voxel above the thresholdcondition, this sub-method stops because there are no voxels satisfyingthe condition. Third, voxels are filled in connecting the new voxel tothe component segment that are within a specified range of voxel values(e.g., above or below the threshold value as described above). Thesethree steps are repeated until there is no voxel in the ball satisfyingthe condition. In other words, an iterative constrained walk from thecomponent segment to the terminal segment of the initial treelikestructure is performed until no voxels satisfy the conditions. Thesegment connection is then attempted again, given this new possiblegrowth. Other methods for connecting disconnected segments back to theinitial segmentation are within the scope of the present invention.

The local extension method as shown and described can be used inimage-guided surgery and navigation, such as described above. Forexample, the method can be used with the imaging, tracking,localization, and navigation systems and methods described in U.S. Ser.Nos. 13/817,730 and 15/290,822, the entireties of which are incorporatedherein by reference. The local extension method described above can beimplemented on a computer using computer processors, memory units,storage devices, computer software, and other components. For example,in one embodiment the local extension method is implemented on thenavigation system 70 (see FIG. 3 ), such as by processor 72. In certainembodiments, navigation system 70 further includes and/or is in datacommunication with the imaging device 40, and the method includesobtaining an image(s) from the image device and performing thesegmentation methods on the image(s). The segmentation methods describedherein can be applied to one or more images from an image dataset, forexample the image datasets as described above. The segmentation methodscan be performed on one or more images before or after the constructionof a fused inspiration-to-expiration model for use in navigation.

In one embodiment, the method is used to locally extend a segmentationin a region of interest to navigate to a nodule or target in an airwaytree using the navigation system 70. For example, FIG. 18 illustrates aninitial segmentation 1200 of an airway tree. The initial segmentation1200 includes the initial treelike structure 1202 and a nodule or target1204 (such as the target tissue 420 described above). In order to accessthe target 1204, the navigation system 70 can determine the pathway 1206(e.g., an initial navigation pathway such as pathway 416 describedabove) through the airway tree to reach the target. After a region ofinterest is identified (e.g., automatically or manually based on userinput such as through I/O component 78) and the disclosed extensionmethod is performed on the initial segmentation 1200, new sub-treecomponents 1208 are identified and connected back to the initialtreelike structure 1202. Based on these new components, the navigationpathway to the target 1204 is changed, because the local extensionmethod found airways offering a more direct or shorter pathway 1210 tothe target (see FIG. 19 ).

In some embodiments, additional information collected during navigationcan be used to enhance the segmentation extension methods describedherein. For example, as described above with reference to step 1100,image data (e.g., 3D image data such as data from CT image(s)) isreceived, and at step 1106, the image data is segmented to create aninitial segmentation. During a navigated procedure, such as thosedescribed above, extension of the segmentation can be enhanced using atleast one of navigation data and navigated image data. Navigation dataincludes data from a 3D localizer or localization element using an EMsensor (e.g., data from a device or instrument tracked by navigationsystem 70, as described in detail above). The navigation data includespoint(s) or a point cloud, such as points corresponding to a traveledpath of the navigated device or instrument, the current location of thenavigated device or instrument, and/or a projected trajectory of thenavigated device or instrument. Navigated image data includes 2D or 3Ddata from a navigated imaging device (e.g., navigated ultrasound). Thenavigated image data includes 2D or 3D image data (e.g., image(s), imagedataset(s)). Points or a point cloud can be identified in the navigatedimage data. The segmentation methods disclosed herein utilize thedifferent sources of information as parameters to determine if thepoint(s) or point cloud from navigation data and navigated image datashould be added to the initial segmentation of the airway structure(such as by modifying the pruning constraints), as described in furtherdetail below.

In one embodiment, the target area for use in extracting the region ofinterest for extension of the segmentation can be defined using theregistered location of a localized medical device tracked by thenavigation system 70 (e.g., steerable catheter 600 or other navigatedmedical device). After the initial segmentation is performed and thenavigated pathway 416 or 1206 is determined, an instrument tracked bythe navigation system 70 is inserted into the airway and navigated to alocation in the airway. The position of the localized instrument isregistered to image space and can be displayed by the navigation system70 as described above. Based on this registered location, a region ofinterest can be defined to extend the initial segmentation as describedwith reference to FIGS. 14-19 . For example, a user can manuallyinitiate an extension of the segmentation based on the current locationof the navigated instrument. Alternatively, the extension of the initialsegmentation can be automatically initiated, for example based on theinstrument approaching a terminal segment of the initial segmentedairway, based on the instrument being positioned in a terminal segmentof the initial segmented airway for a number of consecutive frames, orbased on any other indication that the currently navigated region is ofclinical interest. The region of interest can be centered around thenavigated instrument (e.g., the tip of the navigated instrument), or canbe weighted to extend toward a target area or tissue. In thisembodiment, there is no dependence on a target tissue or nodule; rather,the region of interest or target area is identified based on theposition of a localized instrument. In one embodiment, the segmentationextension methods can be used during a procedure to define a likelihoodthat an airway is in front of a navigated instrument. For example, thecurrent trajectory of a navigated instrument can be used at the end ofan initially segmented airway to define a region of interest, and themethods of FIGS. 14-19 performed to determine if it is likely thatairway structure extends beyond the initial segmentation. Thus, apathway of airway structure can be “built” in front of the navigatedinstrument as a procedure is performed.

In another embodiment, the region of interest for the disclosedextension methods can be defined by a path traveled by a navigatedinstrument. In this embodiment, the segmentation can be extended asdisclosed to give a better representation to the user of a localizedinstrument within an airway. For example, a localized medical device(e.g., steerable catheter 600) tracked by navigation system 70 isnavigated through the airway. If the registered location of thelocalized instrument does not correspond to a part of the initial airwaysegmentation, the navigated path to the registered location can be usedto define the region of interest for segmentation extension. If the pathto the current registered location (e.g., a point cloud defined by thenavigated device) of the localized instrument defines a line or shape ofpoints that match within acceptable criteria, that traveled path orpoint cloud is used to define the region of interest for segmentationextension. The criteria can include, for example, distance from the lineor shape of points of the traveled path to centerlines of the initialsegmentation, compatibility of shape and trajectories of the traveledpath (given by, e.g., principal component analysis) and trajectories ofsegments in the initial segmentation, and/or distance from the traveledpath to a terminal segment of initial segmented airway. If the line orshape of points from the traveled path of the navigated instrumentmatches the initial segmentation as determined by the criteria, abounding box of the points in the traveled path can be used to definethe region of interest for the segmentation extension method.

In one embodiment, a traveled path or point cloud generated by anavigated instrument (e.g., steerable catheter 600 tracked by navigationsystem 70), or a projected trajectory of the navigated instrument, canbe used as a metric or weighted parameter for the pruning step 1144described above. For example, segments/components/portions from theregion growing results can be discarded or kept at step 1144 based onwhether they are compatible with the traveled path or point cloud of anavigated instrument. In one embodiment, if the registered positionsand/or trajectories of the localized instrument match geometrically to apossible component, but the pruning constraints of steps 1146-1152 wouldotherwise discard the component, the pruning constraints can be relaxedbased on the geometric compatibility of the possible component with thenavigated points and trajectories of the localized instrument.Similarly, a traveled path or point cloud generated by a navigatedinstrument can be used to connect components at step 1160.

In one embodiment, the traveled path or point cloud generated by anavigated instrument can be used to determine seed voxels for theconstrained region growing at step 1136. For example, in this embodimentthe detection of highly tubular voxels (e.g., via Hessian analysis) atstep 1134 can be omitted if the navigated instrument creates a pointcloud on a number of consecutive frames that is similar to a narrowcylinder. The centroid of the navigated points can be determined andused as a seed point for the region growing at step 1136. Alternatively,in one embodiment the centroid of the convex hull is determined and usedas a seed point for the region growing at step 1136. The convex hull isthe smallest polyhedron that contains all of the navigated points. Inone embodiment, for example, the centroid of the vertices of the convexhull, which ideally will be a point inside the airway, can be used as aseed point for the region growing step 1136. Other uses for theadditional information obtained during navigation are within the scopeof the present invention.

In one embodiment, the segmentation extension methods as described abovecan be performed for one or more regions of interest in a pre-processingstep. Potential components that are discarded or pruned can beidentified in this pre-processing step. During navigation in aprocedure, the potential points that were discarded or pruned can beadded to the airway if additional navigation information (e.g., based onlocation and/or trajectories of a navigated instrument) indicates thepoints are valid candidates. For example, in one embodiment, an initialsegmentation is performed on image data from inspiration, as describedabove. A target or region of interest can be identified on theinspiration image data, and the segmentation extension methods describedherein performed to extend the segmentation to the target or in theregion of interest. During the segmentation extension, as describedelsewhere herein, certain components, segments, or portions arediscarded during the pruning step based on the criteria analysis. Thesegmented inspiration image data is then deformed to form theinspiration-expiration fused model for navigation. During thispre-processing step, including the segmentation extension, theinformation regarding the data points that were pruned during thesegmentation extension is maintained (e.g., stored in the memory of theimage analysis system and/or the navigation system; stored in externalmemory; stored with the image data, etc.). During navigation, additionalinformation regarding the treelike structure is obtained based on thelocalized position of the navigated instrument and/or additional imagedata collected during the procedure. This additional information can beused to enhance the previous segmentation. For example, certaincomponents may have been pruned or discarded initially, but based on theadditional navigation information those components are more likely to bevalid. The pruning step can be performed again, using the additionalnavigation information to the original information analyzed in the firstpruning. As a result, an enhanced segmentation is obtained withadditional valid components now identified.

As described above, two-dimensional (2D) image datasets may beregistered to three-dimensional (3D) image datasets wherein, the twodimensional image datasets may include, but are not limited to,fluoroscopic images, ultrasound images, etc. and the three-dimensional(3D) image datasets may include, but are not limited, to computedtomography (CT) images, fused computed tomography-positron emissiontomography (CT/PET) images, magnetic resonance imaging (MRI) images. Inone embodiment, point(s) or a point cloud can be obtained from one ormore 2D images during navigation, and those points can be used toenhance the segmentation extension methods described herein. Forexample, a navigated imaging device (e.g., ultrasound) can be usedduring a procedure. The imaging device can be an independent navigateddevice (e.g., including a localization element tracked by the navigationsystem) that is inserted into the airway, or alternatively can be animaging device inserted into the working channel of the steerablecatheter (with either or both of the imaging device and the steerablecatheter including localization elements tracked by the navigationsystem). During navigation of the navigated imaging device, additionalinformation is collected. For example, as described above, the traveledpath of a navigation instrument provides data that can be used toconnect candidate components or to enhance the image segmentation. Inaddition, during navigation the system can acquire a point cloud from 2Dimages from the imaging device. These points can be used to extend thesegmentation in areas other than along the navigated path of the imagingdevice, or to inform the likelihood that certain candidate components,segments, or portions are valid subtrees. For example, certain candidatecomponents that were previously discarded may subsequently be displayedby the navigation system based on additional information gained duringnavigation (e.g., from a navigated instrument (location and/ortrajectory), from imaging data from a navigated imaging device, etc.).

Referring to FIG. 20 , in one embodiment 2D image data captured during aprocedure is used to enhance the segmentation extension methodsdisclosed herein. At step 1300, the position and/or traveled path of anavigated imaging device is tracked. For example, the navigated imagingdevice can comprise an independent navigated device, or an imagingdevice inserted into the working channel of the steerable catheter, asdescribed above. The position and/or traveled path of the navigatedimaging device is tracked by the navigation system 70. The location ofthe tracked imaging device can be displayed, such as by indicia on the3D image data as described above. The 3D image data can comprise aninitial segmentation (such as the initial segmentation discussed aboveat step 1104). The 3D image data can comprise an extension of theinitial segmentation (such as described above at step 1160, whereinvalid components are connected to the initial segmentation). Thelocation of the navigated imaging device is tracked on the 3D imagedata. At step 1302, 2D image data is captured by the navigated imagingdevice. For example, one or more 2D images can be captured by thenavigated imaging device as it is advanced through an airway. The 2Dimage(s) can include image data regarding the airway in which thenavigated imaging device is located, and/or data regarding airwaysadjacent to the location of the navigated imaging device. The 2D imagedata is collected, for example, during a procedure. In one embodiment,the 2D image data comprises 2D ultrasound image(s) obtained during aprocedure. Multiple 2D images can be acquired during the procedure. Atstep 1304, the 2D image data is registered to the 3D image data on whichthe imaging device is being navigated. For example, 2D ultrasound imagedata can be registered to 3D CT image data. At step 1306, a 3D volume isconstructed from the 2D image data gathered during the procedure at step1302. Step 1306 is optional, and it is understood that it can be omittedwithin the scope of the present invention. At step 1308, point(s) or apoint cloud is obtained from the 2D image data (e.g., from the 2Dimage(s) and/or from a 3D volume constructed from the 2D image(s)). Thepoint(s) or point cloud can be obtained manually or automatically. Forexample, in one embodiment a user can manually select (e.g., using I/Ocomponent 78 to select point(s)). In another embodiment, the point(s)can be obtained automatically, such as by segmentation (e.g.,segmentation of a 3D constructed volume from step 1306; segmentation of2D image data from step 1302; etc.). At step 1310, the point(s) obtainedat step 1308 are used to extend the initial segmentation (or furtherextend the already extended initial segmentation), such as in thesegmentation methods described herein above. For example, the pruningconstraints of steps 1146-1152 can be relaxed for certain points thatare identified during step 1308 to display a component that mightotherwise be discarded. Similarly, the point(s) from step 1308 can beused to connect components at step 1160.

The segmentation extension methods as shown and described offer manybenefits over existing image processing methodologies and improve theoperation of image processing software. The described method searchesfor and segments components (e.g., candidate sub-trees) that may bedisconnected from the initial treelike structure in a region ofinterest, and connects the new segments to the previously segmentedtreelike structure based on geometric criteria and morphologicalproperties of the candidate sub-tree. For example, the described methodwill detect potential sub-trees and infer a connection even if there isa mucus cap or motion artifact, which previous segmentation methodscannot do. Previous segmentation methods for finding airways in an imageassume that no bronchi can be detected where no vessels are found.However, the disclosed method does not make this assumption, andtherefore searches regions that may contain airway lumens that previoussegmentation methods would fail to search. For example, finer vessels myfail to survive thresholding after down sampling or due to motion orimaging artifacts, and previous segmentation methods would fail tosearch these areas, for example to limit the time and memory necessaryfor the segmentation, and therefore fail to find airways that thecurrent method would find. The disclosed method extends the segmentationin a region of interest, and therefore reduces the time and processingthat may otherwise be required for segmentation, although the disclosedmethod can be applied iteratively to a coarse airway segmentation togive a more general segmentation method. As shown and described, thedisclosed method region grows at detected seed points and thendetermines whether to discard components based on morphological andshape analysis criteria. These components are not necessarily connectedto the initial segmentation, as in previous segmentation techniques thatmay involve pruning components already attached to region grow results.The disclosed method connects components that may or may not directlyconnect to the prior segmentation based on region growing alone. Themethod uses not only centerline orientation, but also distance andradius estimates, because the components may not connect directly to theprior segmentation. As distinct from previous techniques, the currentmethod does no merging of masks, and only operates on a small sub-regionof the image at a time, and therefore has much lesser memoryrequirements. The region grows are based on several seed points at onceobtained through Hessian analysis of a bilaterally smoothed andcontrast-enhanced sub-image, as opposed to grow regions based on oneseed point alone (e.g., an estimated point in the trachea). Componentsare connected back to the initial segmentation that may not connectdirectly through region growing. The Hessian analysis utilized in themethod imparts information about the orientation of the components,which information is usable in later steps of the method while alsobeing used to discard seed points based on voxel radius estimates.

The segmentation extension methods described herein can be performed onimage(s); image data; image dataset(s); at inspiration, expiration, orother respiratory or anatomical time period; on inspiration image data;on expiration image data; on fused inspiration-expiration image data; orany other suitable image or image data as may be useful for theparticular application. For example, if a region of interest isidentified during expiration, the extension method can be performed onthe expiration image data, or alternatively can be performed on theassociated inspiration image data (e.g., by deforming the region ofinterest back to the inspiration location, performing the extensionmethod, and then deforming back to expiration for further use innavigation). The segmentation extension methods can be performed onimage data before it is used for navigation, or on the image data duringnavigation. Although described herein with reference to airway structureand navigation in the airway structure, it is understood that thesegmentation extension methods can be applied to other anatomicaltreelike structures within the scope of the present invention.

In tests conducted comparing the disclosed extension method to previoussegmentation processes, the disclosed method found additional treelikestructures not previously found. Based on 141 test cases where noduleswere segmented and the disclosed method was applied to an initial airwaysegmentation result, the disclosed method: 1) found on average twoadditional airway lumens in a region of interest; 2) found at least oneadditional airway lumen in a region of interest 73% of the time; 3)improved the distance from the closest airway branch to the nodule 50%of the time; 4) on average, improved the distance from the airway to thenodule surface by 21% because the optimal path to the nodule can changebased on new lumens being discovered; and 5) changed the optimal path tothe nodule via the airway lumens 26% of the time.

It is noted that the terms “comprise” (and any form of comprise, such as“comprises” and “comprising”), “have” (and any form of have, such as“has” and “having”), “contain” (and any form of contain, such as“contains” and “containing”), and “include” (and any form of include,such as “includes” and “including”) are open-ended linking verbs. Thus,a method, an apparatus, or a system that “comprises,” “has,” “contains,”or “includes” one or more items possesses at least those one or moreitems, but is not limited to possessing only those one or more items.Individual elements or steps of the present methods, apparatuses, andsystems are to be treated in the same manner.

The terms “a” and “an” are defined as one or more than one. The term“another” is defined as at least a second or more. The term “coupled”encompasses both direct and indirect connections, and is not limited tomechanical connections.

Those of skill in the art will appreciate that in the detaileddescription above, certain well known components and assembly techniqueshave been omitted so that the present methods, apparatuses, and systemsare not obscured in unnecessary detail.

While various embodiments of the invention have been described above, itshould be understood that they have been presented by way of exampleonly, and not limitation. Thus, the breadth and scope of the inventionshould not be limited by any of the above-described embodiments, butshould be defined only in accordance with the following claims and theirequivalents. Any feature of a disclosed embodiment can be combined withany other feature of any disclosed embodiment within the scope of thepresent invention.

The previous description of the embodiments is provided to enable anyperson skilled in the art to make or use the invention. While theinvention has been particularly shown and described with reference toembodiments thereof, it will be understood by those skilled in art thatvarious changes in form and details may be made therein withoutdeparting from the spirit and scope of the invention.

What is claimed is:
 1. A method of extending a segmentation of an imageusing navigated image data from a navigation system, the methodcomprising: tracking, with the navigation system, at least one of atraveled path and a position of an imaging device relative to an initialsegmentation of 3D image data including an initial treelike structure;capturing, with the navigated imaging device, navigated image datacomprising image data including at least one 2D or 3D image, andnavigation data comprising a point associated with a position andorientation (POSE) of the navigated imaging device when the at least one2D or 3D image is captured; obtaining, by the navigation system, a pointfrom the navigated image data corresponding to a potential airwaystructure; and extending, by the navigation system, the initialsegmentation of 3D image data using the point obtained from thenavigated image data by, extracting a region of interest from the 3Dimage data to create a sub-image; extracting connected components basedon spillage-constrained region growing using seed points from thesub-image; pruning the extracted components to discard components notlikely to be connected to the initial treelike structure and keep onlycandidate components likely to be a valid sub-tree of the initialtreelike structure by, discarding components based on whether they havegenus zero and are relatively convex, including determining whether aratio of a volume of the extracted component before morphologicalclosing to a volume of the extracted component after morphologicalclosing is greater than a threshold; discarding or modifying componentsbased on the components curving more than a threshold amount, includingdetermining whether a dot product of a direction of the extractedcomponent and a direction of a segment of the extracted component isbelow a threshold; discarding components based on a shape analysis ofthe shape of the extracted component, including comparing shape analysismetrics including at least one of a minor axis length, an elongation,and an eccentricity of the extracted component to target metrics; anddiscarding components based on a cylindrical score of the components,including determining a circle equivalent diameter of the extractedcomponent; and connecting the candidate components to the initialtreelike structure.
 2. The method of claim 1, further comprisingregistering the navigated image data to the 3D image data.
 3. The methodof claim 1, further comprising constructing a 3D volume from thenavigated image data.
 4. The method of claim 3, wherein obtaining apoint comprises automatically segmenting the constructed 3D volume toobtain a point from the navigated image data.
 5. The method of claim 1,wherein obtaining a point comprises selecting the point using an I/Ocomponent of the navigation system.
 6. The method of claim 1, whereinobtaining a point comprises obtaining a point cloud.
 7. The method ofclaim 1, wherein extending the initial segmentation using the pointobtained from the navigated image data comprises determining the regionof interest in the 3D image data based on the point obtained from thenavigated image data.
 8. The method of claim 1, wherein extending theinitial segmentation using the point obtained from the navigated imagedata further comprises performing the spillage-constrained regiongrowing using the point obtained from the navigated image data as a seedpoint.
 9. The method of claim 1, wherein extending the initialsegmentation using the point obtained from the navigated image datacomprises relaxing the thresholds and analysis metrics in the pruningstep based on the point obtained from the navigated image data.
 10. Themethod of claim 1, wherein capturing comprises capturing 2D ultrasoundimage data with the imaging device.
 11. A method of extending asegmentation of an image using navigated image data from a navigationsystem, the method comprising: tracking, with the navigation system, atleast one of a traveled path and a position of an imaging devicerelative to an initial segmentation of 3D image data including aninitial treelike structure; capturing, with the navigated imagingdevice, navigated image data comprising image data including at leastone 2D or 3D image, and navigation data comprising a point associatedwith a position and orientation (POSE) of the navigated imaging devicewhen the at least one 2D or 3D image is captured; obtaining, by thenavigation system, a point from the navigated image data correspondingto a potential airway structure; and extending, by the navigationsystem, the initial segmentation of 3D image data using the pointobtained from the navigated image data by, extracting a region ofinterest from the 3D image data to create a sub-image; extractingconnected components based on spillage-constrained region growing usingseed points from the sub-image; pruning the extracted components todiscard components not likely to be connected to the initial treelikestructure and keep only candidate components likely to be a validsub-tree of the initial treelike structure; and connecting the candidatecomponents to the initial treelike structure by, determining whether thecandidate component has boundary voxels that are 4-connected to theinitial segmentation and if so filling in voxels to fully connect thecandidate component to the initial segmentation; determining whether theradius of a candidate component is compatible with a terminal segment ofthe initial segmentation, and filling in voxels to connect the candidatecomponent to the terminal segment if they are compatible; determining ifthe candidate component includes a path of seed points to the initialsegmentation, and if so, identifying potential new growth based on agradient ball method.
 12. The method of claim 11, wherein extending theinitial segmentation using the point obtained from the navigated imagedata comprises relaxing the determinations in the connecting step basedon the point obtained from the navigated image data.
 13. A method ofextending a segmentation of an image using navigation data from anavigation system, the method comprising: tracking, with the navigationsystem, at least one of a traveled path, a position, and a trajectory ofa navigated instrument relative to an initial segmentation of 3D imagedata including an initial treelike structure to create navigation data;and extending the initial segmentation of the 3D image data using thenavigation data, wherein extending the initial segmentation comprises:extracting a region of interest from the 3D image data to create asub-image; extracting connected components based on spillage-constrainedregion growing using seed points from the sub-image; pruning theextracted components to discard components not likely to be connected tothe initial treelike structure and keep only candidate components likelyto be a valid sub-tree of the initial treelike structure; and connectingthe candidate components to the initial treelike structure by,determining whether the candidate component has boundary voxels that are4-connected to the initial segmentation and if so filling in voxels tofully connect the candidate component to the initial segmentation;determining whether the radius of a candidate component is compatiblewith a terminal segment of the initial segmentation, and filling invoxels to connect the candidate component to the terminal segment ifthey are compatible; and determining if the candidate component includesa path of seed points to the initial segmentation, and if so,identifying potential new growth based on a gradient ball method. 14.The method of claim 13, wherein extending the initial segmentationfurther comprises determining a region of interest based on thenavigation data.
 15. The method of claim 13, wherein extending theinitial segmentation further comprises performing thespillage-constrained region growing using the navigation data of thenavigated instrument as seed points.
 16. The method of claim 13, whereinpruning comprises: discarding components based on whether they havegenus zero and are relatively convex, including determining whether aratio of a volume of the extracted component before morphologicalclosing to a volume of the extracted component after morphologicalclosing is greater than a threshold; discarding or modifying componentsbased on the components curving more than a threshold amount, includingdetermining whether a dot product of a direction of the extractedcomponent and a direction of a segment of the extracted component isbelow a threshold; discarding components based on a shape analysis ofthe shape of the extracted component, including comparing shape analysismetrics including at least one of a minor axis length, an elongation,and an eccentricity of the extracted component to target metrics; anddiscarding components based on a cylindrical score of the components,including determining a circle equivalent diameter of the extractedcomponent.
 17. The method of claim 16, wherein extending the initialsegmentation comprises relaxing the thresholds and analysis metrics inthe pruning step based on the navigation data.
 18. The method of claim13, wherein extending the initial segmentation comprises relaxing thedeterminations in the connecting step based on the navigation data.